Computer augmented threat evaluation

ABSTRACT

An automated system attempts to characterize code as safe or unsafe. For intermediate code samples not placed with sufficient confidence in either category, human-readable analysis is automatically generated to assist a human reviewer in reaching a final disposition. For example, a random forest over human-interpretable features may be created and used to identify suspicious features in a manner that is understandable to, and actionable by, a human reviewer. Similarly, a k-nearest neighbor algorithm may be used to identify similar samples of known safe and unsafe code based on a model for, e.g., a file path, a URL, an executable, and so forth. Similar code may then be displayed (with other information) to a user for evaluation in a user interface. This comparative information can improve the speed and accuracy of human interventions by providing richer context for human review of potential threats.

RELATED APPLICATIONS

This application claims the benefit of U.S. Prov. App. No. 62/726,174filed on Aug. 31, 2018, the entire content of which is herebyincorporated by reference.

FIELD

The present disclosure relates to a threat management system.

BACKGROUND

Against a backdrop of continually evolving computer security threats,there remains a need for automated, semi-automated, and manualtechniques to manage security threats to an enterprise network, and toassist with detection, identification, and disposal of potential threatsto the network and network endpoints.

SUMMARY

In one aspect, an ensemble of detection techniques are used to identifycode that presents intermediate levels of threat. For example, anensemble of machine learning techniques may be used to evaluatesuspiciousness based on binaries, file paths, behaviors, reputation andso forth, and code may be sorted into safe, unsafe, and intermediate, orany similar categories. By filtering and prioritizing intermediatethreats with these tools, human threat intervention can advantageouslybe directed toward code samples and associated contexts most appropriatefor non-automated responses.

In another aspect, an automated system attempts to characterize code assafe or unsafe. For intermediate code samples that are not placed withsufficient confidence in either category, human-readable analysis isautomatically generated, such as qualitative or quantitative comparisonsto previously categorized code samples, in order to assist a humanreviewer in reaching a final disposition. For example a random forestover human-interpretable features may be created and used to identifysuspicious features in a manner that is understandable to, andactionable by, a human reviewer. Similarly, a k-nearest neighboralgorithm or similar technique may be used to identify similar samplesof known safe and unsafe code based on a model for one or more of a filepath, a URL, an executable, and so forth. Similar code may then bedisplayed along with other information to a user for evaluation in auser interface. This comparative information can substantially improvethe speed and accuracy of human interventions by providing richercontext for human review of potential threats.

In another aspect, activity on an endpoint is monitored in two stageswith a local agent. In a first stage, particular computing objects onthe endpoint are selected for tracking. In a second stage, particulartypes of changes to those objects are selected. By selecting objects andobject changes in this manner, a compact data stream of informationhighly relevant to threat detection can be provided from an endpoint toa central threat management facility. In order to support dynamic threatresponse, the locus and level of detection applied by the local agentcan be controlled by the threat management facility and/or the endpoint.At the same time, a local data recorder creates a local record of awider range of objects and changes. The system may support forensicactivity by facilitating queries to the local data recorder on theendpoint to retrieve more complete records of local activity when thecompact data stream does not adequately characterize a particularcontext.

In another aspect, in a threat management platform, a number ofendpoints log events in an event data recorder. A local agent filtersthis data and feeds a filtered data stream to a central threatmanagement facility. The central threat management facility can locallyor globally tune filtering by local agents based on the current datastream, and can query local event data recorders for additionalinformation where necessary or helpful in threat detection or forensicanalysis. The central threat management facility also stores and deploysa number of security tools such as a web-based user interface supportedby machine learning models to identify potential threats requiring humanintervention and other models to provide human-readable context forevaluating potential threats.

In another aspect, a computer model is created for automaticallyevaluating the business value of computing objects such as files anddatabases on an endpoint. This can be used to assess the potentialbusiness impact of a security compromise to an endpoint, or a processexecuting on an endpoint, in order to prioritize potential threatswithin an enterprise for human review and intervention.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the devices,systems, and methods described herein will be apparent from thefollowing description of particular embodiments thereof, as illustratedin the accompanying drawings. The drawings are not necessarily to scale,emphasis instead being placed upon illustrating the principles of thedevices, systems, and methods described herein.

FIG. 1 depicts a block diagram of a threat management system.

FIG. 2 depicts a block diagram of a threat management system.

FIG. 3 shows a system for enterprise network threat detection.

FIG. 4 illustrates a threat management system.

FIG. 5 illustrates an event graph stored by a data recorder.

FIG. 6 shows an endpoint recording events with a data recorder.

FIG. 7 shows a flow chart of a method for computer assistedidentification of intermediate threats.

FIG. 8 shows a flow chart of a method for computer augmented threatevaluation.

FIG. 9A shows a user interface for managing intermediate threats in anenterprise network.

FIG. 9B shows a user interface for managing intermediate threats in anenterprise network.

FIG. 10 shows a user interface for managing intermediate threats in anenterprise network.

FIG. 11 shows a flow chart of a method for dynamic filtering of endpointevent streams.

FIG. 12 shows a flow chart of a method for forensic query of local eventstreams in an enterprise network.

FIG. 13 shows a flow chart of a method for threat detection withbusiness impact scoring.

DETAILED DESCRIPTION

Embodiments will now be described with reference to the accompanyingfigures. The foregoing may, however, be embodied in many different formsand should not be construed as limited to the illustrated embodimentsset forth herein.

All documents mentioned herein are hereby incorporated by reference intheir entirety. References to items in the singular should be understoodto include items in the plural, and vice versa, unless explicitly statedotherwise or clear from the text. Grammatical conjunctions are intendedto express any and all disjunctive and conjunctive combinations ofconjoined clauses, sentences, words, and the like, unless otherwisestated or clear from the context. Thus, the term “or” should generallybe understood to mean “and/or” and so forth.

Recitation of ranges of values herein are not intended to be limiting,referring instead individually to any and all values falling within therange, unless otherwise indicated herein, and each separate value withinsuch a range is incorporated into the specification as if it wereindividually recited herein. The words “about,” “approximately” or thelike, when accompanying a numerical value, are to be construed asindicating a deviation as would be appreciated by one of ordinary skillin the art to operate satisfactorily for an intended purpose. Similarly,words of approximation such as “approximately” or “substantially” whenused in reference to physical characteristics, should be understood tocontemplate a range of deviations that would be appreciated by one ofordinary skill in the art to operate satisfactorily for a correspondinguse, function, purpose, or the like. Ranges of values and/or numericvalues are provided herein as examples only, and do not constitute alimitation on the scope of the described embodiments. Where ranges ofvalues are provided, they are also intended to include each value withinthe range as if set forth individually, unless expressly stated to thecontrary. The use of any and all examples, or exemplary language(“e.g.,” “such as,” or the like) provided herein, is intended merely tobetter illuminate the embodiments and does not pose a limitation on thescope of the embodiments. No language in the specification should beconstrued as indicating any unclaimed element as essential to thepractice of the embodiments.

In the following description, it is understood that terms such as“first,” “second,” “top,” “bottom,” “up,” “down,” and the like, arewords of convenience and are not to be construed as limiting terms.

FIG. 1 depicts a block diagram of a threat management system 101providing protection against a plurality of threats, such as malware,viruses, spyware, cryptoware, adware, Trojans, spam, intrusion, policyabuse, improper configuration, vulnerabilities, improper access,uncontrolled access, and more. A threat management facility 100 maycommunicate with, coordinate, and control operation of securityfunctionality at different control points, layers, and levels within thesystem 101. A number of capabilities may be provided by a threatmanagement facility 100, with an overall goal to intelligently use thebreadth and depth of information that is available about the operationand activity of compute instances and networks as well as a variety ofavailable controls. Another overall goal is to provide protection neededby an organization that is dynamic and able to adapt to changes incompute instances and new threats. In embodiments, the threat managementfacility 100 may provide protection from a variety of threats to avariety of compute instances in a variety of locations and networkconfigurations.

Just as one example, users of the threat management facility 100 maydefine and enforce policies that control access to and use of computeinstances, networks and data. Administrators may update policies such asby designating authorized users and conditions for use and access. Thethreat management facility 100 may update and enforce those policies atvarious levels of control that are available, such as by directingcompute instances to control the network traffic that is allowed totraverse firewalls and wireless access points, applications and dataavailable from servers, applications and data permitted to be accessedby endpoints, and network resources and data permitted to be run andused by endpoints. The threat management facility 100 may provide manydifferent services, and policy management may be offered as one of theservices.

Turning to a description of certain capabilities and components of thethreat management system 101, an exemplary enterprise facility 102 maybe or may include any networked computer-based infrastructure. Forexample, the enterprise facility 102 may be corporate, commercial,organizational, educational, governmental, or the like. As home networksget more complicated, and include more compute instances at home and inthe cloud, an enterprise facility 102 may also or instead include apersonal network such as a home or a group of homes. The enterprisefacility's 102 computer network may be distributed amongst a pluralityof physical premises such as buildings on a campus, and located in oneor in a plurality of geographical locations. The configuration of theenterprise facility as shown is merely exemplary, and it will beunderstood that there may be any number of compute instances, less ormore of each type of compute instances, and other types of computeinstances. As shown, the exemplary enterprise facility includes afirewall 10, a wireless access point 11, an endpoint 12, a server 14, amobile device 16, an appliance or IOT device 18, a cloud computinginstance 19, and a server 20. Again, the compute instances 10-20depicted are exemplary, and there may be any number or types of computeinstances 10-20 in a given enterprise facility. For example, in additionto the elements depicted in the enterprise facility 102, there may beone or more gateways, bridges, wired networks, wireless networks,virtual private networks, other compute instances, and so on.

The threat management facility 100 may include certain facilities, suchas a policy management facility 112, security management facility 122,update facility 120, definitions facility 114, network access rulesfacility 124, remedial action facility 128, detection techniquesfacility 130, application protection facility 150, asset classificationfacility 160, entity model facility 162, event collection facility 164,event logging facility 166, analytics facility 168, dynamic policiesfacility 170, identity management facility 172, and marketplacemanagement facility 174, as well as other facilities. For example, theremay be a testing facility, a threat research facility, and otherfacilities. It should be understood that the threat management facility100 may be implemented in whole or in part on a number of differentcompute instances, with some parts of the threat management facility ondifferent compute instances in different locations. For example, some orall of one or more of the various facilities 100, 112-174 may beprovided as part of a security agent S that is included in softwarerunning on a compute instance 10-26 within the enterprise facility. Someor all of one or more of the facilities 100, 112-174 may be provided onthe same physical hardware or logical resource as a gateway, such as afirewall 10, or wireless access point 11. Some or all of one or more ofthe facilities may be provided on one or more cloud servers that areoperated by the enterprise or by a security service provider, such asthe cloud computing instance 109.

In embodiments, a marketplace provider 199 may make available one ormore additional facilities to the enterprise facility 102 via the threatmanagement facility 100. The marketplace provider may communicate withthe threat management facility 100 via the marketplace interfacefacility 174 to provide additional functionality or capabilities to thethreat management facility 100 and compute instances 10-26. Asnon-limiting examples, the marketplace provider 199 may be a third-partyinformation provider, such as a physical security event provider; themarketplace provider 199 may be a system provider, such as a humanresources system provider or a fraud detection system provider; themarketplace provider may be a specialized analytics provider; and so on.The marketplace provider 199, with appropriate permissions andauthorization, may receive and send events, observations, inferences,controls, convictions, policy violations, or other information to thethreat management facility. For example, the marketplace provider 199may subscribe to and receive certain events, and in response, based onthe received events and other events available to the marketplaceprovider 199, send inferences to the marketplace interface, and in turnto the analytics facility 168, which in turn may be used by the securitymanagement facility 122.

The identity provider 158 may be any remote identity management systemor the like configured to communicate with an identity managementfacility 172, e.g., to confirm identity of a user as well as provide orreceive other information about users that may be useful to protectagainst threats. In general, the identity provider may be any system orentity that creates, maintains, and manages identity information forprincipals while providing authentication services to relying partyapplications, e.g., within a federation or distributed network. Theidentity provider may, for example, offer user authentication as aservice, where other applications, such as web applications, outsourcethe user authentication step to a trusted identity provider.

In embodiments, the identity provider 158 may provide user identityinformation, such as multi-factor authentication, to a SaaS application.Centralized identity providers such as Microsoft Azure, may be used byan enterprise facility instead of maintaining separate identityinformation for each application or group of applications, and as acentralized point for integrating multifactor authentication. Inembodiments, the identity management facility 172 may communicatehygiene, or security risk information, to the identity provider 158. Theidentity management facility 172 may determine a risk score for a userbased on the events, observations, and inferences about that user andthe compute instances associated with the user. If a user is perceivedas risky, the identity management facility 172 can inform the identityprovider 158, and the identity provider 158 may take steps to addressthe potential risk, such as to confirm the identity of the user, confirmthat the user has approved the SaaS application access, remediate theuser's system, or such other steps as may be useful.

In embodiments, threat protection provided by the threat managementfacility 100 may extend beyond the network boundaries of the enterprisefacility 102 to include clients (or client facilities) such as anendpoint 22 outside the enterprise facility 102, a mobile device 26, acloud computing instance 109, or any other devices, services or the likethat use network connectivity not directly associated with or controlledby the enterprise facility 102, such as a mobile network, a public cloudnetwork, or a wireless network at a hotel or coffee shop. While threatsmay come from a variety of sources, such as from network threats,physical proximity threats, secondary location threats, the computeinstances 10-26 may be protected from threats even when a computeinstance 10-26 is not connected to the enterprise facility 102 network,such as when compute instances 22, 26 use a network that is outside ofthe enterprise facility 102 and separated from the enterprise facility102, e.g., by a gateway, a public network, and so forth.

In some implementations, compute instances 10-26 may communicate withcloud applications, such as a SaaS application 156. The SaaS application156 may be an application that is used by but not operated by theenterprise facility 102. Exemplary commercially available SaaSapplications 156 include Salesforce, Amazon Web Services (AWS)applications, Google Apps applications, Microsoft Office 365applications and so on. A given SaaS application 156 may communicatewith an identity provider 158 to verify user identity consistent withthe requirements of the enterprise facility 102. The compute instances10-26 may communicate with an unprotected server (not shown) such as aweb site or a third-party application through an internetwork 154 suchas the Internet or any other public network, private network orcombination of these.

In embodiments, aspects of the threat management facility 100 may beprovided as a stand-alone solution. In other embodiments, aspects of thethreat management facility 100 may be integrated into a third-partyproduct. An application programming interface (e.g. a source codeinterface) may be provided such that aspects of the threat managementfacility 100 may be integrated into or used by or with otherapplications. For instance, the threat management facility 100 may bestand-alone in that it provides direct threat protection to anenterprise or computer resource, where protection is subscribed todirectly 100. Alternatively, the threat management facility may offerprotection indirectly, through a third-party product, where anenterprise may subscribe to services through the third-party product,and threat protection to the enterprise may be provided by the threatmanagement facility 100 through the third-party product.

The security management facility 122 may provide protection from avariety of threats by providing, as non-limiting examples, endpointsecurity and control, email security and control, web security andcontrol, reputation-based filtering, machine learning classification,control of unauthorized users, control of guest and non-compliantcomputers, and more.

The security management facility 122 may provide malicious codeprotection to a compute instance. The security management facility 122may include functionality to scan applications, files, and data formalicious code, remove or quarantine applications and files, preventcertain actions, perform remedial actions, as well as other securitymeasures. Scanning may use any of a variety of techniques, includingwithout limitation signatures, identities, classifiers, and othersuitable scanning techniques. In embodiments, the scanning may includescanning some or all files on a periodic basis, scanning an applicationwhen the application is executed, scanning data transmitted to or from adevice, scanning in response to predetermined actions or combinations ofactions, and so forth. The scanning of applications, files, and data maybe performed to detect known or unknown malicious code or unwantedapplications. Aspects of the malicious code protection may be provided,for example, in the security agent of an endpoint 12, in a wirelessaccess point 11 or firewall 10, as part of application protection 150provided by the cloud, and so on.

In an embodiment, the security management facility 122 may provide foremail security and control, for example to target spam, viruses, spywareand phishing, to control email content, and the like. Email security andcontrol may protect against inbound and outbound threats, protect emailinfrastructure, prevent data leakage, provide spam filtering, and more.Aspects of the email security and control may be provided, for example,in the security agent of an endpoint 12, in a wireless access point 11or firewall 10, as part of application protection 150 provided by thecloud, and so on.

In an embodiment, security management facility 122 may provide for websecurity and control, for example, to detect or block viruses, spyware,malware, unwanted applications, help control web browsing, and the like,which may provide comprehensive web access control enabling safe,productive web browsing. Web security and control may provide Internetuse policies, reporting on suspect compute instances, security andcontent filtering, active monitoring of network traffic, URI filtering,and the like. Aspects of the web security and control may be provided,for example, in the security agent of an endpoint 12, in a wirelessaccess point 11 or firewall 10, as part of application protection 150provided by the cloud, and so on.

In an embodiment, the security management facility 122 may provide fornetwork access control, which generally controls access to and use ofnetwork connections. Network control may stop unauthorized, guest, ornon-compliant systems from accessing networks, and may control networktraffic that is not otherwise controlled at the client level. Inaddition, network access control may control access to virtual privatenetworks (VPN), where VPNs may, for example, include communicationsnetworks tunneled through other networks and establishing logicalconnections acting as virtual networks. In embodiments, a VPN may betreated in the same manner as a physical network. Aspects of networkaccess control may be provided, for example, in the security agent of anendpoint 12, in a wireless access point 11 or firewall 10, as part ofapplication protection 150 provided by the cloud, e.g., from the threatmanagement facility 100 or other network resource(s).

In an embodiment, the security management facility 122 may provide forhost intrusion prevention through behavioral monitoring and/or runtimemonitoring, which may guard against unknown threats by analyzingapplication behavior before or as an application runs. This may includemonitoring code behavior, application programming interface calls madeto libraries or to the operating system, or otherwise monitoringapplication activities. Monitored activities may include, for example,reading and writing to memory, reading and writing to disk, networkcommunication, process interaction, and so on. Behavior and runtimemonitoring may intervene if code is deemed to be acting in a manner thatis suspicious or malicious. Aspects of behavior and runtime monitoringmay be provided, for example, in the security agent of an endpoint 12,in a wireless access point 11 or firewall 10, as part of applicationprotection 150 provided by the cloud, and so on.

In an embodiment, the security management facility 122 may provide forreputation filtering, which may target or identify sources of knownmalware. For instance, reputation filtering may include lists of URIs ofknown sources of malware or known suspicious IP addresses, code authors,code signers, or domains, that when detected may invoke an action by thethreat management facility 100. Based on reputation, potential threatsources may be blocked, quarantined, restricted, monitored, or somecombination of these, before an exchange of data can be made. Aspects ofreputation filtering may be provided, for example, in the security agentof an endpoint 12, in a wireless access point 11 or firewall 10, as partof application protection 150 provided by the cloud, and so on. Inembodiments, some reputation information may be stored on a computeinstance 10-26, and other reputation data available through cloudlookups to an application protection lookup database, such as may beprovided by application protection 150.

In embodiments, information may be sent from the enterprise facility 102to a third party, such as a security vendor, or the like, which may leadto improved performance of the threat management facility 100. Ingeneral, feedback may be useful for any aspect of threat detection. Forexample, the types, times, and number of virus interactions that anenterprise facility 102 experiences may provide useful information forthe preventions of future virus threats. Feedback may also be associatedwith behaviors of individuals within the enterprise, such as beingassociated with most common violations of policy, network access,unauthorized application loading, unauthorized external device use, andthe like. In embodiments, feedback may enable the evaluation orprofiling of client actions that are violations of policy that mayprovide a predictive model for the improvement of enterprise policies.

An update management facility 120 may provide control over when updatesare performed. The updates may be automatically transmitted, manuallytransmitted, or some combination of these. Updates may include software,definitions, reputations or other code or data that may be useful to thevarious facilities. For example, the update facility 120 may managereceiving updates from a provider, distribution of updates to enterprisefacility 102 networks and compute instances, or the like. Inembodiments, updates may be provided to the enterprise facility's 102network, where one or more compute instances on the enterprisefacility's 102 network may distribute updates to other computeinstances.

The threat management facility 100 may include a policy managementfacility 112 that manages rules or policies for the enterprise facility102. Exemplary rules include access permissions associated withnetworks, applications, compute instances, users, content, data, and thelike. The policy management facility 112 may use a database, a textfile, other data store, or a combination to store policies. In anembodiment, a policy database may include a block list, a black list, anallowed list, a white list, and more. As a few non-limiting examples,policies may include a list of enterprise facility 102 external networklocations/applications that may or may not be accessed by computeinstances, a list of types/classifications of network locations orapplications that may or may not be accessed by compute instances, andcontextual rules to evaluate whether the lists apply. For example, theremay be a rule that does not permit access to sporting websites. When awebsite is requested by the client facility, a security managementfacility 122 may access the rules within a policy facility to determineif the requested access is related to a sporting website.

The policy management facility 112 may include access rules and policiesthat are distributed to maintain control of access by the computeinstances 10-26 to network resources. Exemplary policies may be definedfor an enterprise facility, application type, subset of applicationcapabilities, organization hierarchy, compute instance type, user type,network location, time of day, connection type, or any other suitabledefinition. Policies may be maintained through the threat managementfacility 100, in association with a third party, or the like. Forexample, a policy may restrict instant messaging (IM) activity bylimiting such activity to support personnel when communicating withcustomers. More generally, this may allow communication for departmentsas necessary or helpful for department functions, but may otherwisepreserve network bandwidth for other activities by restricting the useof IM to personnel that need access for a specific purpose. In anembodiment, the policy management facility 112 may be a stand-aloneapplication, may be part of the network server facility 142, may be partof the enterprise facility 102 network, may be part of the clientfacility, or any suitable combination of these.

The policy management facility 112 may include dynamic policies that usecontextual or other information to make security decisions. As describedherein, the dynamic policies facility 170 may generate policiesdynamically based on observations and inferences made by the analyticsfacility. The dynamic policies generated by the dynamic policy facility170 may be provided by the policy management facility 112 to thesecurity management facility 122 for enforcement.

In embodiments, the threat management facility 100 may provideconfiguration management as an aspect of the policy management facility112, the security management facility 122, or some combination.Configuration management may define acceptable or requiredconfigurations for the compute instances 10-26, applications, operatingsystems, hardware, or other assets, and manage changes to theseconfigurations. Assessment of a configuration may be made againststandard configuration policies, detection of configuration changes,remediation of improper configurations, application of newconfigurations, and so on. An enterprise facility may have a set ofstandard configuration rules and policies for particular computeinstances which may represent a desired state of the compute instance.For example, on a given compute instance 12, 14, 18, a version of aclient firewall may be required to be running and installed. If therequired version is installed but in a disabled state, the policyviolation may prevent access to data or network resources. A remediationmay be to enable the firewall. In another example, a configurationpolicy may disallow the use of USB disks, and policy management 112 mayrequire a configuration that turns off USB drive access via a registrykey of a compute instance. Aspects of configuration management may beprovided, for example, in the security agent of an endpoint 12, in awireless access point 11 or firewall 10, as part of applicationprotection 150 provided by the cloud, or any combination of these.

In embodiments, the threat management facility 100 may also provide forthe isolation or removal of certain applications that are not desired ormay interfere with the operation of a compute instance 10-26 or thethreat management facility 100, even if such application is not malwareper se. The operation of such products may be considered a configurationviolation. The removal of such products may be initiated automaticallywhenever such products are detected, or access to data and networkresources may be restricted when they are installed and running. In thecase where such applications are services which are provided indirectlythrough a third-party product, the applicable application or processesmay be suspended until action is taken to remove or disable thethird-party product.

The policy management facility 112 may also require update management(e.g., as provided by the update facility 120). Update management forthe security facility 122 and policy management facility 112 may beprovided directly by the threat management facility 100, or, forexample, by a hosted system. In embodiments, the threat managementfacility 100 may also provide for patch management, where a patch may bean update to an operating system, an application, a system tool, or thelike, where one of the reasons for the patch is to reduce vulnerabilityto threats.

In embodiments, the security facility 122 and policy management facility112 may push information to the enterprise facility 102 network and/orthe compute instances 10-26, the enterprise facility 102 network and/orcompute instances 10-26 may pull information from the security facility122 and policy management facility 112, or there may be a combination ofpushing and pulling of information. For example, the enterprise facility102 network and/or compute instances 10-26 may pull update informationfrom the security facility 122 and policy management facility 112 viathe update facility 120, an update request may be based on a timeperiod, by a certain time, by a date, on demand, or the like. In anotherexample, the security facility 122 and policy management facility 112may push the information to the enterprise facility's 102 network and/orcompute instances 10-26 by providing notification that there are updatesavailable for download and/or transmitting the information. In anembodiment, the policy management facility 112 and the security facility122 may work in concert with the update management facility 120 toprovide information to the enterprise facility's 102 network and/orcompute instances 10-26. In various embodiments, policy updates,security updates and other updates may be provided by the same ordifferent modules, which may be the same or separate from a securityagent running on one of the compute instances 10-26.

As threats are identified and characterized, the definition facility 114of the threat management facility 100 may manage definitions used todetect and remediate threats. For example, identity definitions may beused for scanning files, applications, data streams, etc. for thedetermination of malicious code. Identity definitions may includeinstructions and data that can be parsed and acted upon for recognizingfeatures of known or potentially malicious code. Definitions also mayinclude, for example, code or data to be used in a classifier, such as aneural network or other classifier that may be trained using machinelearning. Updated code or data may be used by the classifier to classifythreats. In embodiments, the threat management facility 100 and thecompute instances 10-26 may be provided with new definitionsperiodically to include most recent threats. Updating of definitions maybe managed by the update facility 120, and may be performed upon requestfrom one of the compute instances 10-26, upon a push, or somecombination. Updates may be performed upon a time period, on demand froma device 10-26, upon determination of an important new definition or anumber of definitions, and so on.

A threat research facility (not shown) may provide a continuouslyongoing effort to maintain the threat protection capabilities of thethreat management facility 100 in light of continuous generation of newor evolved forms of malware. Threat research may be provided byresearchers and analysts working on known threats, in the form ofpolicies, definitions, remedial actions, and so on.

The security management facility 122 may scan an outgoing file andverify that the outgoing file is permitted to be transmitted accordingto policies. By checking outgoing files, the security managementfacility 122 may be able discover threats that were not detected on oneof the compute instances 10-26, or policy violation, such transmittal ofinformation that should not be communicated unencrypted.

The threat management facility 100 may control access to the enterprisefacility 102 networks. A network access facility 124 may restrict accessto certain applications, networks, files, printers, servers, databases,and so on. In addition, the network access facility 124 may restrictuser access under certain conditions, such as the user's location, usagehistory, need to know, job position, connection type, time of day,method of authentication, client-system configuration, or the like.Network access policies may be provided by the policy managementfacility 112, and may be developed by the enterprise facility 102, orpre-packaged by a supplier. Network access facility 124 may determine ifa given compute instance 10-22 should be granted access to a requestednetwork location, e.g., inside or outside of the enterprise facility102. Network access facility 124 may determine if a compute instance 22,26 such as a device outside the enterprise facility 102 may access theenterprise facility 102. For example, in some cases, the policies mayrequire that when certain policy violations are detected, certainnetwork access is denied. The network access facility 124 maycommunicate remedial actions that are necessary or helpful to bring adevice back into compliance with policy as described below with respectto the remedial action facility 128. Aspects of the network accessfacility 124 may be provided, for example, in the security agent of theendpoint 12, in a wireless access point 11, in a firewall 10, as part ofapplication protection 150 provided by the cloud, and so on.

In an embodiment, the network access facility 124 may have access topolicies that include one or more of a block list, a black list, anallowed list, a white list, an unacceptable network site database, anacceptable network site database, a network site reputation database, orthe like of network access locations that may or may not be accessed bythe client facility. Additionally, the network access facility 124 mayuse rule evaluation to parse network access requests and apply policies.The network access rule facility 124 may have a generic set of policiesfor all compute instances, such as denying access to certain types ofwebsites, controlling instant messenger accesses, or the like. Ruleevaluation may include regular expression rule evaluation, or other ruleevaluation method(s) for interpreting the network access request andcomparing the interpretation to established rules for network access.Classifiers may be used, such as neural network classifiers or otherclassifiers that may be trained by machine learning.

The threat management facility 100 may include an asset classificationfacility 160. The asset classification facility will discover the assetspresent in the enterprise facility 102. A compute instance such as anyof the compute instances 10-26 described herein may be characterized asa stack of assets. The one level asset is an item of physical hardware.The compute instance may be, or may be implemented on physical hardware,and may have or may not have a hypervisor, or may be an asset managed bya hypervisor. The compute instance may have an operating system (e.g.,Windows, MacOS, Linux, Android, iOS). The compute instance may have oneor more layers of containers. The compute instance may have one or moreapplications, which may be native applications, e.g., for a physicalasset or virtual machine, or running in containers within a computingenvironment on a physical asset or virtual machine, and thoseapplications may link libraries or other code or the like, e.g., for auser interface, cryptography, communications, device drivers,mathematical or analytical functions and so forth. The stack may alsointeract with data. The stack may also or instead interact with users,and so users may be considered assets.

The threat management facility may include entity models 162. The entitymodels may be used, for example, to determine the events that aregenerated by assets. For example, some operating systems may provideuseful information for detecting or identifying events. For examples,operating systems may provide process and usage information thataccessed through an API. As another example, it may be possible toinstrument certain containers to monitor the activity of applicationsrunning on them. As another example, entity models for users may defineroles, groups, permitted activities and other attributes.

The event collection facility 164 may be used to collect events from anyof a wide variety of sensors that may provide relevant events from anasset, such as sensors on any of the compute instances 10-26, theapplication protection facility 150, a cloud computing instance 109 andso on. The events that may be collected may be determined by the entitymodels. There may be a variety of events collected. Events may include,for example, events generated by the enterprise facility 102 or thecompute instances 10-26, such as by monitoring streaming data through agateway such as firewall 10 and wireless access point 11, monitoringactivity of compute instances, monitoring stored files/data on thecompute instances 10-26 such as desktop computers, laptop computers,other mobile computing devices, and cloud computing instances 19, 109.Events may range in granularity. An exemplary event may be communicationof a specific packet over the network. Another exemplary event may beidentification of an application that is communicating over a network.

The event logging facility 166 may be used to store events collected bythe event collection facility 164. The event logging facility 166 maystore collected events so that they can be accessed and analyzed by theanalytics facility 168. Some events may be collected locally, and someevents may be communicated to an event store in a central location orcloud facility. Events may be logged in any suitable format.

Events collected by the event logging facility 166 may be used by theanalytics facility 168 to make inferences and observations about theevents. These observations and inferences may be used as part ofpolicies enforced by the security management facility Observations orinferences about events may also be logged by the event logging facility166.

When a threat or other policy violation is detected by the securitymanagement facility 122, the remedial action facility 128 may be used toremediate the threat. Remedial action may take a variety of forms,non-limiting examples including collecting additional data about thethreat, terminating or modifying an ongoing process or interaction,sending a warning to a user or administrator, downloading a data filewith commands, definitions, instructions, or the like to remediate thethreat, requesting additional information from the requesting device,such as the application that initiated the activity of interest,executing a program or application to remediate against a threat orviolation, increasing telemetry or recording interactions for subsequentevaluation, (continuing to) block requests to a particular networklocation or locations, scanning a requesting application or device,quarantine of a requesting application or the device, isolation of therequesting application or the device, deployment of a sandbox, blockingaccess to resources, e.g., a USB port, or other remedial actions. Moregenerally, the remedial action facility 122 may take any steps or deployany measures suitable for addressing a detection of a threat, potentialthreat, policy violation or other event, code or activity that mightcompromise security of a computing instance 10-26 or the enterprisefacility 102.

FIG. 2 depicts a block diagram of a threat management system 201 such asany of the threat management systems described herein, and including acloud enterprise facility 280. The cloud enterprise facility 280 mayinclude servers 284, 286, and a firewall 282. The servers 284, 286 onthe cloud enterprise facility 280 may run one or more enterpriseapplications and make them available to the enterprise facilities 102compute instances 10-26. It should be understood that there may be anynumber of servers 284, 286 and firewalls 282, as well as other computeinstances in a given cloud enterprise facility 280. It also should beunderstood that a given enterprise facility may use both SaaSapplications 156 and cloud enterprise facilities 280, or, for example, aSaaS application 156 may be deployed on a cloud enterprise facility 280.As such, the configurations in FIG. 1 and FIG. 2 are shown by way ofexamples and not exclusive alternatives.

FIG. 3 shows a system 300 for enterprise network threat detection. Thesystem 300 may use any of the various tools and techniques for threatmanagement contemplated herein. In the system, a number of endpointssuch as the endpoint 302 may log events in a data recorder 304. A localagent on the endpoint 302 such as the security agent 306 may filter thisdata and feeds a filtered data stream to a threat management facility308 such as a central threat management facility or any of the otherthreat management facilities described herein. The threat managementfacility 308 can locally or globally tune filtering by local agentsbased on the current data stream, and can query local event datarecorders for additional information where necessary or helpful inthreat detection or forensic analysis. The threat management facility308 may also or instead store and deploys a number of security toolssuch as a web-based user interface that is supported by machine learningmodels to aid in the identification and assessment of potential threatsby a human user. This may, for example, include machine learninganalysis of new code samples, models to provide human-readable contextfor evaluating potential threats, and any of the other tools ortechniques described herein. More generally, the threat managementfacility 308 may provide any of a variety of threat management tools 316to aid in the detection, evaluation, and remediation of threats orpotential threats.

The threat management facility 308 may perform a range of threatmanagement functions such as any of those described herein. The threatmanagement facility 308 may generally include an application programminginterface 310 to third party services 320, a user interface 312 foraccess to threat management and network administration functions, and anumber of threat detection tools 314.

In general, the application programming interface 310 may supportprogrammatic connections with third party services 320. The applicationprogramming interface 310 may, for example, connect to Active Directoryor other customer information about files, data storage, identities anduser profiles, roles, access privileges and so forth. More generally theapplication programming interface 310 may provide a programmaticinterface for customer or other third party context, information,administration and security tools, and so forth. The applicationprogramming interface 310 may also or instead provide a programmaticinterface for hosted applications, identity provider integration toolsor services, and so forth.

The user interface 312 may include a website or other graphicalinterface or the like, and may generally provide an interface for userinteraction with the threat management facility 308, e.g., for threatdetection, network administration, audit, configuration and so forth.This user interface 312 may generally facilitate human curation ofintermediate threats as contemplated herein, e.g., by presentingintermediate threats along with other supplemental information, andproviding controls for user to dispose of such intermediate threats asdesired, e.g., by permitting execution or access, by denying executionor access, or by engaging in remedial measures such as sandboxing,quarantining, vaccinating, and so forth.

The threat detection tools 314 may be any of the threat detection tools,algorithms, techniques or the like described herein, or any other toolsor the like useful for detecting threats or potential threats within anenterprise network. This may, for example, include signature basedtools, behavioral tools, machine learning models, and so forth. Ingeneral, the threat detection tools 314 may use event data provided byendpoints within the enterprise network, as well as any other availablecontext such as network activity, heartbeats, and so forth to detectmalicious software or potentially unsafe conditions for a network orendpoints connected to the network. In one aspect, the threat detectiontools 314 may usefully integrate event data from a number of endpoints(including, e.g., network components such as gateways, routers andfirewalls) for improved threat detection in the context of complex ordistributed threats. The threat detection tools 314 may also or insteadinclude tools for reporting to a separate modeling and analysis platform318, e.g., to support further investigation of security issues, creationor refinement of threat detection models or algorithms, review andanalysis of security breaches and so forth.

The threat management tools 316 may generally be used to manage orremediate threats to the enterprise network that have been identifiedwith the threat detection tools 314 or otherwise. Threat managementtools 316 may, for example, include tools for sandboxing, quarantining,removing, or otherwise remediating or managing malicious code ormalicious activity, e.g., using any of the techniques described herein.

The endpoint 302 may be any of the endpoints or other compute instancesor the like described herein. This may, for example, include end-usercomputing devices, mobile devices, firewalls, gateways, servers, routersand any other computing devices or instances that might connect to anenterprise network. As described above, the endpoint 302 may generallyinclude a security agent 306 that locally supports threat management onthe endpoint 302, such as by monitoring for malicious activity, managingsecurity components on the endpoint 302, maintaining policy compliance,and communicating with the threat management facility 308 to supportintegrated security protection as contemplated herein. The securityagent 306 may, for example, coordinate instrumentation of the endpoint302 to detect various event types involving various computing objects onthe endpoint 302, and supervise logging of events in a data recorder304. The security agent 306 may also or instead scan computing objectssuch as electronic communications or files, monitor behavior ofcomputing objects such as executables, and so forth. The security agent306 may, for example, apply signature-based or behavioral threatdetection techniques, machine learning models (e.g. models developed bythe modeling and analysis platform), or any other tools or the likesuitable for detecting malware or potential malware on the endpoint 302.

The data recorder 304 may log events occurring on or related to theendpoint. This may, for example, include events associated withcomputing objects on the endpoint 302 such as file manipulations,software installations, and so forth. This may also or instead includeactivities directed from the endpoint 302, such as requests for contentfrom Uniform Resource Locators or other network activity involvingremote resources. The data recorder 304 may record data at any frequencyand any level of granularity consistent with proper operation of theendpoint 302 in an intended or desired manner.

The endpoint 302 may include a filter 322 to manage a flow ofinformation from the data recorder 304 to a remote resource such as thethreat detection tools 314 of the threat management facility 308. Inthis manner, a detailed log of events may be maintained locally on eachendpoint, while network resources can be conserved for reporting of afiltered event stream that contains information believed to be mostrelevant to threat detection. The filter 322 may also or instead beconfigured to report causal information that causally relatescollections of events to one another. In general, the filter 322 may beconfigurable so that, for example, the threat management facility 308can increase or decrease the level of reporting based on a currentsecurity status of the endpoint, a group of endpoints, the enterprisenetwork and the like. The level of reporting may also or instead bebased on currently available network and computing resources, or anyother appropriate context.

In another aspect, the endpoint 302 may include a query interface 324 sothat remote resources such as the threat management facility 308 canquery the data recorder 304 remotely for additional information. Thismay include a request for specific events, activity for specificcomputing objects, or events over a specific time frame, or somecombination of these. Thus for example, the threat management facility308 may request all changes to the registry of system information forthe past forty eight hours, all files opened by system processes in thepast day, all network connections or network communications within thepast hour, or any other parametrized request for activities monitored bythe data recorder 304. In another aspect, the entire data log, or theentire log over some predetermined window of time, may be request forfurther analysis at a remote resource.

It will be appreciated that communications among third party services320, a threat management facility 308, and one or more endpoints such asthe endpoint 302 may be facilitated by using consistent namingconventions across products and machines. For example, the system 300may usefully implement globally unique device identifiers, useridentifiers, application identifiers, data identifiers, Uniform ResourceLocators, network flows, and files. The system may also or instead usetuples to uniquely identify communications or network connections basedon, e.g., source and destination addresses and so forth.

According to the foregoing, a system disclosed herein includes anenterprise network, and endpoint coupled to the enterprise network, anda threat management facility coupled in a communicating relationshipwith the endpoint and a plurality of other endpoints through theenterprise network. The endpoint may have a data recorder that stores anevent stream of event data for computing objects, a filter for creatinga filtered event stream with a subset of event data from the eventstream, and a query interface for receiving queries to the data recorderfrom a remote resource, the endpoint further including a local securityagent configured to detect malware on the endpoint based on event datastored by the data recorder, and further configured to communicate thefiltered event stream over the enterprise network. The threat managementfacility may be configured to receive the filtered event stream from theendpoint, detect malware on the endpoint based on the filtered eventstream, and remediate the endpoint when malware is detected, the threatmanagement facility further configured to modify security functionswithin the enterprise network based on a security state of the endpoint.

The threat management facility may be configured to adjust reporting ofevent data through the filter in response to a change in the filteredevent stream received from the endpoint. The threat management facilitymay be configured to adjust reporting of event data through the filterwhen the filtered event stream indicates a compromised security state ofthe endpoint. The threat management facility may be configured to adjustreporting of event data from one or more other endpoints in response toa change in the filtered event stream received from the endpoint. Thethreat management facility may be configured to adjust reporting ofevent data through the filter when the filtered event stream indicates acompromised security state of the endpoint. The threat managementfacility may be configured to request additional data from the datarecorder when the filtered event stream indicates a compromised securitystate of the endpoint. The threat management facility may be configuredto request additional data from the data recorder when a security agentof the endpoint reports a security compromise independently from thefiltered event stream. The threat management facility may be configuredto adjust handling of network traffic at a gateway to the enterprisenetwork in response to a predetermined change in the filtered eventstream. The threat management facility may include a machine learningmodel for identifying potentially malicious activity on the endpointbased on the filtered event stream. The threat management facility maybe configured to detect potentially malicious activity based on aplurality of filtered event streams from a plurality of endpoints. Thethreat management facility may be configured to detect malware on theendpoint based on the filtered event stream and additional context forthe endpoint.

The data recorder may record one or more events from a kernel driver.The data recorder may record at least one change to a registry of systemsettings for the endpoint. The endpoints may include a server, afirewall for the enterprise network, a gateway for the enterprisenetwork, or any combination of these. The endpoint may be coupled to theenterprise network through a virtual private network or a wirelessnetwork. The endpoint may be configured to periodically transmit asnapshot of aggregated, unfiltered data from the data recorder to thethreat management facility for remote storage. The data recorder may beconfigured to delete records in the data recorder corresponding to thesnapshot in order to free memory on the endpoint for additionalrecording.

FIG. 4 illustrates a threat management system. In general, the systemmay include an endpoint 402, a firewall 404, a server 406 and a threatmanagement facility 408 coupled to one another directly or indirectlythrough a data network 405, all as generally described above. Each ofthe entities depicted in FIG. 4 may, for example, be implemented on oneor more computing devices such as the computing device described herein.A number of systems may be distributed across these various componentsto support threat detection, such as a coloring system 410, a keymanagement system 412 and a heartbeat system 414, each of which mayinclude software components executing on any of the foregoing systemcomponents, and each of which may communicate with the threat managementfacility 408 and an endpoint threat detection agent 420 executing on theendpoint 402 to support improved threat detection and remediation.

The coloring system 410 may be used to label or color software objectsfor improved tracking and detection of potentially harmful activity. Thecoloring system 410 may, for example, label files, executables,processes, network communications, data sources and so forth with anysuitable information. A variety of techniques may be used to selectstatic and/or dynamic labels for any of these various software objects,and to manage the mechanics of applying and propagating coloringinformation as appropriate. For example, a process may inherit a colorfrom an application that launches the process. Similarly, a file mayinherit a color from a process when it is created or opened by aprocess, and/or a process may inherit a color from a file that theprocess has opened. More generally, any type of labeling, as well asrules for propagating, inheriting, changing, or otherwise manipulatingsuch labels, may be used by the coloring system 410 as contemplatedherein.

The key management system 412 may support management of keys for theendpoint 402 in order to selectively permit or prevent access to contenton the endpoint 402 on a file-specific basis, a process-specific basis,an application-specific basis, a user-specific basis, or any othersuitable basis in order to prevent data leakage, and in order to supportmore fine-grained and immediate control over access to content on theendpoint 402 when a security compromise is detected. Thus, for example,if a particular process executing on the endpoint is compromised, orpotentially compromised or otherwise under suspicion, keys to thatprocess may be revoked in order to prevent, e.g., data leakage or othermalicious activity.

The heartbeat system 414 may be used to provide periodic or aperiodicinformation from the endpoint 402 or other system components aboutsystem health, security, status, and so forth. A heartbeat may beencrypted or plaintext, or some combination of these, and may becommunicated unidirectionally (e.g., from the endpoint 408 to the threatmanagement facility 408) or bidirectionally (e.g., between the endpoint402 and the server 406, or any other pair of system components) on anyuseful schedule.

In general, these various monitoring and management systems maycooperate to provide improved threat detection and response. Forexample, the coloring system 410 may be used to evaluate when aparticular process is potentially opening inappropriate files based onan inconsistency or mismatch in colors, and a potential threat may beconfirmed based on an interrupted heartbeat from the heartbeat system414. The key management system 412 may then be deployed to revoke keysto the process so that no further files can be opened, deleted orotherwise modified. More generally, the cooperation of these systemsenables a wide variety of reactive measures that can improve detectionand remediation of potential threats to an endpoint.

FIG. 5 illustrates an event graph 500 stored by a data recorder such asany of the data recorders described herein. The event graph 500 mayinclude a sequence of computing objects causally related by a number ofevents, and which provide a description of computing activity on one ormore endpoints. The event graph 500 may be generated, for example, whena security event 502 is detected on an endpoint, and may be based on adata log or similar records obtained by an event data recorder duringoperation of the endpoint. The event graph 500 may be used to determinea root cause 504 of the security event 502 as generally described above.The event graph 500 may also or instead be continuously generated toserve as, or be a part of, the data log obtained by the data recorder.In any case, an event graph 500, or a portion of an event graph 500 in awindow before or around the time of a security event, may be obtainedand analyzed after a security event 502 occurs to assist in determiningits root cause 504. The event graph 500 depicted in the figure isprovided by way of example only, and it will be understood that manyother forms and contents for event graphs 500 are also or insteadpossible. It also will be understood that while the figure illustrates agraphical depiction of an event graph 500, the event graph 500 may bestored in any suitable data structure or combination of data structuressuitable for capturing the chain of events and objects in a manner thatpreserves causal relationships for use in forensics and malwaredetection as contemplated herein.

By way of example, the event graph 500 depicted in the figure beginswith a computing object that is a USB device 512, which may be connectedto an endpoint. Where the USB device 512 includes a directory or filesystem, the USB device 512 may be mounted or accessed by a file systemon an endpoint to read contents. The USB device 512 may be detected 513and contents of the USB device 512 may be opened 514, e.g., by a user ofthe endpoint or automatically by the endpoint in response to detectionof the USB device 512. The USB device 512 may include one or more filesand applications, e.g., a first file 516, a second file 518, and a firstapplication 520. The first file 516 may be associated with a first event522 and the second file may be associated with a second event 524. Thefirst application 520 may access one or more files on the endpoint,e.g., the third file 526 shown in the figure. The first application 520may also or instead perform one or more actions 528, such as accessing aURL 530. Accessing the URL 530 may download or run a second application532 on the endpoint, which in turn accesses one or more files (e.g., thefourth file 534 shown in the figure) or is associated with other events(e.g., the third event 536 shown in the figure).

In the example provided by the event graph 500 depicted in the figure,the detected security event 502 may include the action 528 associatedwith the first application 520, e.g., accessing the URL 530. By way ofexample, the URL 530 may be a known malicious URL or a URL or networkaddress otherwise associated with malware. The URL 530 may also orinstead include a blacklisted network address that although notassociated with malware may be prohibited by a security policy of theendpoint or enterprise network in which the endpoint is a participant.The URL 530 may have a determined reputation or an unknown reputation.Thus, accessing the URL 530 can be detected through known computingsecurity techniques.

In response to detecting the security event 502, the event graph 500 maybe traversed in a reverse order from a computing object associated withthe security event 502 based on the sequence of events included in theevent graph 500. For example, traversing backward from the action 528leads to at least the first application 520 and the USB device 512. Aspart of a root cause analysis, one or more cause identification rulesmay be applied to one or more of the preceding computing objects havinga causal relationship with the detected security event 502, or to eachcomputing object having a causal relationship to another computingobject in the sequence of events preceding the detected security event502. For example, other computing objects and events may be tangentiallyassociated with causally related computing objects when traversing theevent graph 500 in a reverse order—such as the first file 516, thesecond file 518, the third file 525, the first event 522, and the secondevent 524 depicted in the figure. In an aspect, the one or more causeidentification rules are applied to computing objects preceding thedetected security event 502 until a cause of the security event 502 isidentified.

In the example shown in the figure, the USB device 512 may be identifiedas the root cause 504 of the security event 502. In other words, the USBdevice 512 was the source of the application (the first application 520)that initiated the security event 502 (the action 528 of accessing thepotentially malicious or otherwise unwanted URL 530).

The event graph 500 may similarly be traversed going forward from one ormore of the root cause 504 or the security event 502 to identify one ormore other computing objects affected by the root cause 504 or thesecurity event 502. For example, the first file 516 and the second 518potentially may be corrupted because the USB device 512 includedmalicious content. Similarly, any related actions performed after thesecurity event 502 such as any performed by the second application 532may be corrupted. Further testing or remediation techniques may beapplied to any of the computing objects affected by the root cause 504or the security event 502.

The event graph 500 may include one or more computing objects or eventsthat are not located on a path between the security event 502 and theroot cause 504. These computing objects or events may be filtered or‘pruned’ from the event graph 500 when performing a root cause analysisor an analysis to identify other computing objects affected by the rootcause 504 or the security event 502. For example, computing objects orevents that may be pruned from the event graph 500 may include the USBdrive 510 and the USB device being detected 513.

It will be appreciated that the event graph 500 depicted in FIG. 5 is anabstracted, simplified version of actual nodes and events on an endpointfor demonstration. Numerous other nodes and edges will be present in aworking computing environment. For example, when a USB device is coupledto an endpoint, the new hardware will first be detected, and then theendpoint may search for suitable drivers and, where appropriate, presenta user inquiry of how the new hardware should be handled. A user maythen apply a file system to view contents of the USB device and select afile to open or execute as desired, or an autorun.exe or similar filemay be present on the USB device that begins to execute automaticallywhen the USB device is inserted. All of these operations may requiremultiple operating system calls, file system accesses, hardwareabstraction layer interaction, and so forth, all of which may bediscretely represented within the event graph 500, or abstracted up to asingle event or object as appropriate. Thus, it will be appreciated thatthe event graph 500 depicted in the drawing is intended to serve as anillustrative example only, and not to express or imply a particularlevel of abstraction that is necessary or useful for root causeidentification as contemplated herein.

The event graph 500 may be created or analyzed using rules that defineone or more relationships between events and computing objects. The CLanguage Integrated Production System (CLIPS) is a public domainsoftware tool intended for building expert systems, and may be suitablyadapted for analysis of a graph such as the event graph 500 to identifypatterns and otherwise apply rules for analysis thereof. While othertools and programming environments may also or instead be employed,CLIPS can support a forward and reverse chaining inference enginesuitable for a large amount of input data with a relatively small set ofinference rules. Using CLIPS, a feed of new data can trigger a newinference, which may be suitable for dynamic solutions to root causeinvestigations.

An event graph such as the event graph 500 shown in the figure mayinclude any number of nodes and edges, where computing objects arerepresented by nodes and events are represented by edges that mark thecausal or otherwise directional relationships between computing objectssuch as data flows, control flows, network flows and so forth. Whileprocesses or files are common forms of nodes that might appear in such agraph, any other computing object such as an IP address, a registry key,a domain name, a uniform resource locator, a command line input or otherobject may also or instead be designated to be a node in an event graphas contemplated herein. Similarly, while an edge may be formed by an IPconnection, a file read, a file write, a process invocation (parent,child, etc.), a process path, a thread injection, a registry write, adomain name service query, a uniform resource locator access and soforth other edges may be designated. As described above, when a securityevent is detected, the source of the security event may serve as astarting point within the event graph 500, which may then be traversedbackward to identify a root cause using any number of suitable causeidentification rules. The event graph 500 may then usefully be traversedforward from that root cause to identify other computing objects thatare potentially tainted by the root cause so that a more completeremediation can be performed.

FIG. 6 shows an endpoint recording events with a data recorder. Thesystem 600 may include an endpoint 610 containing a data recorder 620, amonitoring facility 630, and any number of objects 612 and events 614.An analysis facility 640 may be coupled in a communicating relationshipwith the endpoint 610 over a data network 650 such as any of thenetworks described above. It will be appreciated that, while illustratedas components of the endpoint 610, certain components of the system 600such as the data recorder 620 and the monitoring facility 630 and theanalysis facility may also or instead be realized as remote servicesinstantiated on a virtual appliance, a public or private cloud, or thelike, any of which may be coupled to the endpoint 610 through the datanetwork 650 or another communication channel (not shown). Each of thecomponents of the system 600 may be configured with suitable programmingand configuration to participate in the various forensic techniques,threat detection techniques, and security management techniquescontemplated herein.

The endpoint 610 may be any of the endpoints described herein, e.g., acomputing device in an enterprise network, or any other device ornetwork asset that might join or participate in an enterprise orotherwise operate on an enterprise network. This may, for example,include a server, a client device such as a desktop computer or a mobilecomputing device (e.g., a laptop computer or a tablet), a cellularphone, a smart phone, or other computing device suitable forparticipating in the system 600 or in an enterprise.

In general, the endpoint 610 may include any number of computing objects612, which may for example, be processes executed by one or moreprocessors or other processing circuitry, files or data stored inmemory, or any other computing objects described herein. While the termobject has a number of specific meanings in the art, and in particularin object-oriented programming, it will be understood that the term‘object’ as used herein is intended to be significantly broader, and mayinclude any data, process, file or combination of these includingwithout limitation any process, application, executable, script, dynamiclinked library (DLL), file, data, database, data source, data structure,function, resource locator (e.g., uniform resource locator (URL) orother uniform resource identifier (URI)), or the like that might beresident on the endpoint 610 and manipulated by the endpoint 610 oranother component of the system 600 or other systems described elsewhereherein. The object 612 may also or instead include a remote resource,such as a resource identified in a URL. That is, while the object 612 inthe figure is depicted as residing on the endpoint 610, an object 612may also reside elsewhere in the system 600, and may be specified forexample with a link, pointer, or reference that is locally stored on theendpoint 610.

The object 612 may be an item that is performing an action or causing anevent 614, or the object 612 may be an item that is receiving the actionor is the result of an event 614 (e.g., the object 612 may be an item inthe system 600 being acted upon by an event 614 or another object 612).In general, an event 614 as contemplated herein may be any data flow,execution flow, control flow, network flow, or other similar action orevent that might causally relate objects 612 to one another. Where theobject 612 is data or includes data, the object 612 may be encrypted orotherwise protected, or the object 612 may be unencrypted or otherwiseunprotected. The object 612 may be a process or other computing objectthat performs an action, which may include a single event 614 or acollection or sequence of events 614 taken by a process. The object 612may also or instead include an item such as a file or lines of code thatare executable to perform such actions. The object 612 may also orinstead include a computing component upon which an action is taken,e.g., a system setting (e.g., a registry key or the like), a data file,a URL, and so forth. The object 612 may exhibit a behavior such as aninteraction with another object or a component of the system 600.

Objects 612 may be described in terms of persistence. The object 612may, for example, be a part of a process, and remain persistent as longas that process is alive. The object 612 may instead be persistentacross an endpoint 610 and remain persistent as long as an endpoint 610is active or alive. The object 612 may instead be a global object havingpersistence outside of an endpoint 610, such as a URL or a data store.In other words, the object 612 may be a persistent object withpersistence outside of the endpoint 610.

Although many if not most objects 612 will typically be benign objectsforming a normal part of the computing environment for an operatingendpoint 610, an object 612 may contain software associated with anadvanced persistent threat (APT) or other malware that resides partiallyor entirely on the endpoint 610. This associated software may havereached the endpoint 610 in a variety of ways, and may have been placedmanually or automatically on the endpoint 610 by a malicious source. Itwill be understood that the associated software may take any number offorms and have any number of components. For example, the associatedsoftware may include an executable file that can execute independently,or the associated software may be a macro, plug-in, or the like thatexecutes within another application. Similarly, the associated softwaremay manifest as one or more processes or threads executing on theendpoint 610. Further, the associated software may install from a fileon the endpoint 610 (or a file remote from the endpoint 610), and theassociated software may create one or more files such as data files orthe like while executing. Associated software should be understood togenerally include all such files and processes except where a specificfile or process is more specifically noted.

An event 614 may include an action, a behavior, an interaction, and soforth. The event 614 may be generated by or otherwise related to anobject 612. For example, the event 614 may be associated with a file andinclude an action such as a read, a write, an open, a move, a copy, adelete, and so forth. The event 614 may also or instead include aninter-process communication, e.g., a create, a handle, a debug, a remoteinjection, and so forth. The event 614 may also or instead include anetwork action such as accessing an Internet Protocol (IP) address orURL. It will also be understood that the event 614 may be, e.g., akernel-level event, a software-level event, a hardware-level or devicedriver event, a communications event, a file system event and so forth.In another aspect, the event 614 may be a synthetic event that is basedon a combination of other discrete events, or based on a score, metric(e.g., reputation) or other calculated or derived quantity orevaluation, as well as any combination of the foregoing. Thus, whileevents are illustrated as isolated discrete items in FIG. 6, events maybe compound items, calculated items, analytical results, and so forth.In one aspect, created synthetic or derivative events that are formed ofcombinations of other events or calculated metrics and the like mayusefully compress the amount of storage required for the data recorder620, and or the amount of network communications required to reportpotentially relevant events to a remote resource such as the analysisfacility 640.

The data recorder 620 may monitor and record activity related to theobjects 612 and events 614 occurring on the endpoint 610. The activityof the endpoint 610 may be stored in a data log 622 or the like on thedata recorder 620, which may be stored locally on the endpoint 610 (asdepicted) or remotely at a threat management resource, or somecombination of these, such as where the data log 622 is periodicallytransmitted to a remote facility for archiving or analysis. The datarecorder 620 may continuously record any activity occurring on theendpoint 610 for predetermined periods of time before overwritingpreviously recorded data. Thus, the data log 622 may include acontinuous data feed of events 614. When an event 614 is detected thatis a beacon or trigger event (such as a file detection, a malicioustraffic detection, or the like), the data log 622 may be saved andtransmitted to an analysis facility 640 or the like for analysis, e.g.,to determine a root cause of the beacon or trigger event. The data log622 may be used to create an event graph or other snapshot of theactivity on the endpoint 610, e.g., for a period of time surrounding abeacon or trigger event. The beacon or trigger event may be detectedlocally by the monitoring facility 630, or remotely by a remote threatmanagement facility or the like, or some combination of these.

While illustrated on the endpoint 610, it will be understood that thedata recorder 620 may also or instead be implemented at a remotelocation such as a threat management facility or other enterprisenetwork security resource, or some combination of these. The datarecorder 620 may be provisioned on the same or a different device than adata store in which data is stored. The data recorder 620 may beconfigured to record data as efficiently as possible so as to minimizeimpact on the endpoint 610. It will further be appreciated that, while asingle data recorder is depicted, the endpoint 610 may include anynumber of data recorders, which may operate independently or in acoordinated manner, e.g., to distribute logging functions or prioritizemonitoring of highly sensitive hardware or software. Furthermore,multiple endpoints may contain data records that report asynchronouslyor in a coordinated manner to the analysis facility 640.

The monitoring facility 630 may work in conjunction with the datarecorder 620 to instrument the endpoint 610 so that any observableevents 614 by or involving various objects 612 can be monitored andrecorded. It will be appreciated that various filtering rules andtechniques may be used to synopsize, summarize, filter, compress orotherwise process information captured by the data recorder 620 to helpensure that relevant information is captured while maintaining practicallimits on the amount of information that is gathered.

A security product 632 may execute on the endpoint 610 to detect asecurity event on the endpoint 610, which may act as the beacon ortrigger event for the system 600. The security product 632 may usetechniques such as signature-based and behavioral-based malwaredetection including without limitation one or more of host intrusionprevention, malicious traffic detection, URL blocking, file-baseddetection, and so forth.

The beacon or trigger event on the endpoint 610 may be a fully qualified(e.g., definitive) detection of a compromise or other maliciousactivity. In another aspect, the beacon or trigger event on the endpoint610 may be a suspicious behavior that is suspicious but not confirmed asmalicious. For example, the beacon or trigger event on the endpoint 610may signal an unusual behavior that is known to commonly appearconcurrently with the detection of malware. In an aspect, when thebeacon or trigger event is a suspicious behavior, the data log 622 maybe analyzed differently than when the beacon or trigger event is aconfirmed malicious behavior. For example, the data log 622 may be sentto a different component of the system 600 through the network, e.g., toa different analysis facility 640.

The monitoring facility 630 may be disposed remotely from the endpoint610 or analysis facility 640. The monitoring facility 630 may beincluded on one or more of the endpoint 610 or analysis facility 640. Inan aspect, the monitoring facility 630 and the analysis facility 640included in the same component.

The analysis facility 640 may analyze the data log 622, e.g., as part ofa root cause analysis and to identify objects 612 compromised by theroot cause. To this end, the analysis facility 640 may utilize one ormore rules 642 for applying to the data included in the data log 622 todetermine a root cause of a beacon or trigger event such as a suspectedor actual security compromise on the endpoint 610. The analysis facility640 may reside locally on the endpoint 610 (e.g., be a part of, embeddedwithin, or locally coupled to the endpoint 610). The analysis facility640 may be an external facility, or it may reside in a virtual appliance(e.g., which could be run by a protected set of systems on their ownnetwork systems), a private cloud, a public cloud, and so forth. Theanalysis facility 640 may store locally-derived threat information foruse in subsequent identification, remediation, or other similaractivity. The analysis facility 640 may also or instead receive threatinformation from a third-party source such as any public, private,educational, or other organization that gathers information on networkthreats and provides analysis and threat detection information for useby others. This third-party information may, for example, be used toimprove detection rules or other forensic analysis that might beperformed on information in the data log 622.

The analysis facility 640 may create an event graph. In general, theevent graph may represent information in the data log 622 in a graphwhere objects 612 are nodes and events 614 are edges connecting thenodes to one another based on causal or other relationships as generallycontemplated herein. The event graph may be used by the analysisfacility 640 or other component(s) of the system 600 as part of a rootcause analysis and to identify objects 612 compromised by the rootcause. The event graph may also or instead be displayed to a user of thesystem 600 or endpoint 610, e.g., using an interactive user interface orthe like. In one aspect, the analysis facility 640 may be incorporatedinto a threat management facility for an enterprise network. In anotheraspect, the analysis facility 640 may operate independently from thethreat management facility and may be, for example, a third party,remote service or the like.

The system 600 may advantageously use the data log 622 to configure andinitialize an analysis in a sandboxed or otherwise isolated environmentwhere the execution of the recorded activity related to a detectedsecurity event is allowed to run. That is, rather than uploading acomplete image of an endpoint 610 using conventional techniques, thedata log 622 may include only a series of events/processes related tothe detected event that may be uploaded for execution/analysis. Theanalysis may thus include executing this series of events/processes inthe same order to determine a threat level for the endpoint 610.

The data log 622 may include data from a single endpoint 610, or from anumber of endpoints 610, for example where one endpoint 610 accesses aservice or a file on another endpoint. This advantageously facilitatestracking or detection of potentially malicious activity that spansmultiple devices, particularly where the behavior on a single endpointdoes not appear malicious. Thus, the monitoring facility 630 may monitoractivity from an endpoint 610 exclusively, or use the full context ofactivity from all protected endpoints 610, or some combination of these.Similarly, the event graph generated from the data log 622 may includeactivity from one endpoint 610 exclusively, or use the full context ofactivity from all protected endpoints 610, or some combination of these.Data logs 622 and event graphs may also or instead be stored for futureanalyses, e.g., for comparing to future data logs and event graphs.

Similarly, the events may include human interactions such as keyboardstrokes, mouse clicks or other input and output to human interfacedevices and hardware. This usefully permits discrimination within causalchains among events initiated by processes executing on a device andevents that are initiated or controlled by a human user that is presenton the endpoint.

In one aspect, the data recorder 620 may monitor events from a low-leveldriver, referred to herein as an endpoint defense driver, installed inthe kernel space early in an operating system installation or bootprocess, e.g., prior to population of the user space with userapplications and the like. In this manner, the endpoint defense drivermay be configured to instrument operation of the endpoint so that fileoperations and interprocess communications are passed through the kernelwhere the endpoint defense driver can enforce restrictions on filemodifications, code injections and so forth, and provide visibility oversuch interprocess communications for purposes of recording event data ascontemplated herein. Certain related techniques are described, forexample, in U.S. patent application Ser. No. 15/795,952, filed on Oct.27, 2017, the entire contents of which are hereby incorporated byreference. As described therein, in order to secure interprocesscommunications and ensure that protected computing objects are notmodified, each process can be configured to communicate with otherprocesses using a system call that passes through the kernel space, andin particular the endpoint defense driver, in order to ensure that theendpoint defense driver has an opportunity to regulate process activityin a manner consistent with a list of protected objects maintained in aprotection cache maintained in the kernel.

The endpoint defense driver may maintain a number of caches to assist inmonitoring the endpoint, such as a process cache, a protection cache(also referred to as a protected object cache or tamper protectioncache), and a file cache. In general, the process cache may storeinformation related to a process such as the application name,application family (e.g., a vendor or commonly used name for a suite ofsoftware including installers, libraries, supporting applications orprocesses, and so forth), an application path, and an applicationcategory (such as any of the categories or types described herein). Theprotection cache may support tamper protection tools. In particular, theendpoint defense driver may initially load a list of protected objectssuch as registry keys, services, applications, directories, and soforth. The endpoint defense driver may proactively prevent any changesto these protected objects (which include the protection cache itself),or may prevent any changes except by other protected objects identifiedin the protection cache. The file cache may contain information aboutfiles on the endpoint, and may store any useful information includinginformation about protection status, modifications, local or globalreputation, and so forth. The endpoint defense driver can use thesecaches in a variety of ways to support secure operation of an endpointprotection system. For example, as noted above, by directinginterprocess communications and file system operations through theendpoint defense driver, security and tamper prevention can be ensuredon an object-by-object basis, e.g., for registry keys, files, processes,directories, and so forth. The endpoint defense driver can also set andretrieve information about new processes as they are launched in theuser space. A data recorder may usefully record transactions detectedby, or managed by or through, the endpoint defense driver, which mayadvantageously provide secure, kernel-level monitoring of processesexecuting on an endpoint.

FIG. 7 shows a flow chart of a method for computer assistedidentification of intermediate threats. In general, an ensemble ofdetection techniques are used to identify potential threats that presentintermediate levels of threat. For example, an ensemble of machinelearning techniques may be used to evaluate suspiciousness based onbinaries, file paths, behaviors, reputation, and so forth, and potentialthreats may be sorted into safe, unsafe, and intermediate, or anysimilar categories. By filtering and prioritizing intermediate threatswith these tools, human threat intervention can advantageously bedirected toward threat samples and associated contexts most appropriatefor non-automated responses.

As shown in step 702, the method 700 may include providing a trainingset including threat samples that are known to be safe and known to bemalicious. This may, for example, include a random or curated trainingset of malicious and safe code, behaviors, user actions, networkactivities, messaging content, and so forth, or any combination of theforegoing. The training set may usefully be updated periodically as newsample of, e.g., known safe and known unsafe code are positivelyidentified by a threat management facility or a third party securityservice or the like.

As shown in step 704, the method 700 may include tagging each one of thethreat samples with one or more tags that identify corresponding,observed behavior. This may, for example, include automatic tagging ofthreat samples based on models of known behavior, code, and so forth, orthis may include clustering or other unsupervised machine learninganalysis. Tagging may also or instead include human sorting and taggingaccording to empirical observations of behavior relevant or potentiallyrelevant to security. This may also or instead include human sorting andcurating of machine-assigned tags. Tags may identify malware types(e.g., spyware, adware, advanced persistent threat, ransomware, and soforth) or general behavioral characteristics (unpacker). In one aspect,these semantic tags may be assigned continuous values based on relativesimilarity to one or more known semantic types. This information may beused, e.g., in training to provide hints about the likely nature ofunknown threat samples with similar features. The resulting tags may beused when training models, and may advantageously permit a neuralnetwork or other machine learning model to simultaneously draw multipleinferences about a new threat sample.

As shown in step 706, the method 700 may include training models forthreat identification or evaluation. For example, this may includetraining a first machine learning model to identify malicious code inthe training set based on the one or more tags, or otherwise trainingthe machine learning model to identify code with malicious behaviorusing a training set including threat samples that are known to be safeand threat samples that are known to be malicious. Other machinelearning models and techniques may also or instead be usefully createdto support an ensemble machine learning approach to threat detection andanalysis. Thus, for example, this may include training a second machinelearning model to identify threats in the training set based on acorresponding file path for each of the threat samples, or otherwisetraining the second machine learning model to identify malicious orpotentially malicious code based on a file path using a training setincluding threat samples that are known to be safe and threat samplesthat are known to be malicious. This may also or instead includetraining a third machine learning model to identify malicious code inthe training set based on one or more Uniform Resource Locators (orother network addresses, remote resource identifiers, or the like)contained in a threat sample, or otherwise training the third model toidentify potential threat samples based on a Uniform Resource Locator orthe like found in a training set including threat samples that are knownto be safe and threat samples that are known to be malicious.

As shown in step 708, the method 700 may include creating an integrativemodel 710 that evaluates a probability that an unknown threat sample ismalicious based on a combination of the first machine learning model,the second machine learning model, and the third machine learning model.For example, this may include creating an integrative model 710 thatevaluates a potential threat by a threat sample based on a combinationof a first model configured to identify malicious code based onbehavioral tags, a second model configured to identify malicious codebased on an executable file path, and a third model configured toidentify malicious code based on a Uniform Resource Locator within thethreat sample or accessed by the threat sample, or any combination ofthese and/or any other machine learning models or the like. In oneaspect, the integrative model 710 may also generally evaluate potentialthreats based on a context for a threat sample. For example, the contextmay include a reputation for the threat sample, a user executing aprocess associated with the threat sample, one or more files accessed bythe threat sample, or any other context or other information availableto the integrative model 710 and useful for assessing potential threats.

More generally, any other information may be used in combination withthe ensemble of machine learning techniques described above as necessaryor helpful to improve estimates of riskiness. Further, other machinelearning models may be used in addition to or instead of the machinelearning models described above, to the extent that such model(s) can betrained to accurately or usefully estimate risk. Thus, for example, if amachine learning model can be trained to accurately identify threatsbased on, e.g., access control lists, certificates, signatures, hashes,communications protocols, content, and so forth. Further, it may beadvantageous to design and apply a group of machine learning models withdetection techniques that are generally uncorrelated to one another inorder to provide greater sensitivity to various types of threats.

As shown in step 712, the method 700 may include identifyingintermediate threats. Threats samples such as executables that are knownto be safe or known to be unsafe can be easily disposed ofautomatically. Similarly, threat samples that are very likely safe orunsafe, e.g., within a predetermined threshold of likelihood, cantypically be safely disposed of without human intervention. However,intermediate threats—threats that are not within a predeterminedlikelihood of being safe or unsafe—present significant challenges tomachine learning models that rely more on pattern matching than causalrelationships or explanations to discriminate among safe and unsafethreat samples. Thus, the method 700 contemplated herein can usefullyidentify intermediate threats and elevate these items for humanintervention. This may, for example, include determining if a new threatsample is an intermediate threat, such as a threat that fails to fallwithin a first predetermined threshold of likely safe or within a secondpredetermined threshold of likely malicious based on a probabilitycalculated by the integrative model. In another aspect, this may includeidentifying a new threat sample (or other threat sample) as anintermediate threat when the new threat sample is not within apredetermined confidence level of safe code or malicious code accordingto the integrative model.

It will be understood that a threat sample, as used herein, may includeany of a variety of samples suitable for assessing actual or potentialthreats to an enterprise network. For example, a threat sample mayinclude a computing object such as executable code in the form ofcompiled code or object code, or executing code including withoutlimitation a code sample, a process, a script, an executable or othercode sample. This may also instead include any other computing objectsuitable for, e.g., a behavioral analysis, a signature analysis, anevaluation by a machine learning model, or the like. In another aspect,a threat sample may include a data structure or the like such as a file,cache, registry, or other data repository. The threat sample may also orinstead include events such as actions by code, changes to data, accessto remote network resources, receipt or transmittal of electroniccommunications, uploads or downloads of data, connections to devicessuch as USB drives, user interactions through a user interface, or anyother events or the like that can be instrumented and monitored on anendpoint. These various types of threat samples may be used alone or inany combination to assist in detecting and evaluating intermediatethreats as contemplated herein.

As shown in step 714, the method 700 may include ranking theintermediate threats identified in step 714. This may include a rankingbased on an estimated suspiciousness or threat based on, e.g., thelikelihood of being safe or unsafe according to the integrative model710, the number of similar threat samples that are known to be safe orunsafe, and/or any other context relevant to evaluating the potentialthreat posed by a threat sample. More generally, any technique forprioritizing unknown threat samples so that a human user can directattention toward potentially riskier items may usefully be employed torank the intermediate threats as contemplated herein. The rankedintermediate threats may be organized into a list or other datastructure or the like for subsequent display to a user.

As shown in step 716, the method 700 may include displaying intermediatethreats for user disposition. For example, this may includeconditionally presenting a new threat sample for human intervention whenthe probability calculated by the integrative model identifies the newthreat sample as an intermediate threat, and/or when the threat sampleis ranked sufficiently high relative to other intermediate threats asdescribed above. Displaying the intermediate threats may includeproviding a user interface for presenting the new threat sample with theintermediate threat for human evaluation. This may also or insteadinclude displaying a plurality of intermediate threats, each failing tofall within the first predetermined threshold and the secondpredetermined threshold, in a user interface. As described above, theplurality of intermediate threats may be ranked according to likelihoodof threat, or any other metric or combination of metrics such assimilarity to known malicious code, other indicia of malware or otherthreats, and so forth. Thus, the user interface may present the newthreat sample in a list of a number of intermediate threats detected onan endpoint (or more generally in an enterprise network) and rankedaccording to a likelihood of threat. In one aspect, the plurality ofintermediate threats may be ranked according to a combination oflikelihood of threat and estimated business value (e.g., of one or morefiles associated with each of the intermediate threats) so that humanintervention can also or instead be directed toward items posing agreater economic risk to an enterprise.

As shown in step 718, the method 700 may also include disposing ofintermediate threats, such as through user interactions with informationpresented in the user interface. In one aspect, the user interface mayinclude one or more tools for receiving a user evaluation of one of thethreat samples that presents the intermediate threat. This may also orinstead include one or more tools for remediating a threat associatedwith the threat sample.

According to the foregoing, there is also disclosed herein a system forcomputer assisted identification of intermediate threats. The system mayinclude a memory storing an integrative model configured to evaluate apotential threat by a threat sample based on a combination of a firstmodel configured to identify malicious code based on behavioral tags, asecond model configured to identify malicious code based on anexecutable file path, and a third model configured to identify maliciouscode based on a Uniform Resource Locator within the threat sample. Thesystem may also include a threat management facility configured to applythe integrative model to a new threat sample and to identify a newthreat sample as an intermediate threat. The system may also include aweb server configured to display the intermediate threat in a userinterface on an endpoint for evaluation. The web server may also orinstead be configured to present additional contextual information forthe intermediate threat to a user through the user interface. The webserver may also or instead be configured to receive an evaluation of theintermediate threat from a user through the user interface.

FIG. 8 shows a flow chart of a method for computer augmented threatevaluation. In general, an automated system attempts to characterizecode as safe or unsafe. For intermediate threat samples that are notplaced with sufficient confidence in either category, human-readableanalysis is automatically generated, such as qualitative or quantitativecomparisons to previously categorized threat samples, in order to assista human reviewer in reaching a final disposition. For example a randomforest over human-interpretable features may be created and used toidentify suspicious features in a manner that is understandable to, andactionable by, a human reviewer. Similarly, a k-nearest neighboralgorithm or similar technique may be used to identify similar samplesof known safe and unsafe code based on a model for one or more of a filepath, a URL, an executable, and so forth. Similar code may then bedisplayed along with other information to a user for evaluation in auser interface. This comparative information can substantially improvethe speed and accuracy of human interventions by providing richercontext for human review of potential threats.

As shown in step 802, the method 800 may include providing a model suchas a threat detection model for evaluating a likelihood that a threatsample is at least one of safe or malicious based on a training set ofknown threat samples. This may include any of the machine learningmodels or other threat detection models contemplated herein. As shown instep 804, the method 800 may also include providing threat samples suchas samples of code that are known to be safe and samples of code thatare known to be malicious. This may also or instead include known safeand unsafe samples of network activity, file content, file activity,behaviors, events, and so forth. The threat detection model may includea machine learning model trained using these threat samples, or anyother suitable training set, or some combination of these. Thus,providing the model may include training a machine learning model toidentify malicious code in a training set including threat samples thatare known to be safe and known to be malicious.

The model may include a model for evaluating a likelihood that a threatsample is at least one of safe or malicious based on a training set ofknown threat samples. The model may also or instead include anintegrative model that evaluates a potential threat by a threat samplebased on a combination of a first model configured to identify maliciouscode based on behavioral tags, a second model configured to identifymalicious code based on an executable file path, and a third modelconfigured to identify malicious code based on a Uniform ResourceLocator within the threat sample, or any of the other integrative modelscontemplated herein.

As shown in step 806, the method 800 may include identifyingintermediate threats. For example, this may include identifying a newthreat sample as an intermediate threat that is not within apredetermined likelihood of being malicious or safe according to themodel, or using any of the other techniques described herein.

As shown in step 808, the method 800 may include identifyingsupplemental information relevant to evaluation of the new threatsample, such as relevant features of the new threat sample contributingto an inference of malicious code.

For example, the method 800 may include identifying one or morefeatures, such as relevant features of the new threat sample associatedwith an inference of malicious code, using a random forest overhuman-interpretable features associated with an inference of maliciouscode in the training set of known threat samples (or any other suitabletraining set or the like). Random forests or random decision forests arean ensemble learning method for classification, regression and othertasks, that operate by constructing a multitude of decision trees attraining time and outputting the class that is the mode of the classes(classification) or mean prediction (regression) of the individualtrees. As a significant advantage, the structure of the decision tree(s)can be organized around human-interpretable features such as whether athreat sample is signed or whether the threat sample opens new filesduring execution. While the creation of a random forest is generallycomputationally expensive, and other more efficient techniques are knownfor automated classification, the output of a random forest overhuman-interpretable features can provide highly useful context to ahuman reviewer when evaluating intermediate threats as contemplatedherein, and thus provides particular advantages over otherclassification techniques in this context, even when used in addition toother (possibly more computationally efficient) classification modelsand techniques for evaluating riskiness of unknown threat samples.

Identifying supplemental information may also or instead includeidentifying similar threat samples known to be safe or maliciousincluding one or more safe threat samples similar to the new threatsample and one or more malicious threat samples similar to the newthreat sample. In this context, similarity may usefully be computedbased on a k-nearest neighbor algorithm. The similar threat samples may,for example, include a list of safe threat samples ranked based onsimilarity to the new threat sample according to the k-nearest neighboralgorithm, which may in turn be presented as a ranked list in a userinterface. The similar code may also or instead include a list ofmalicious threat samples ranked based on similarity to the new threatsample according to the k-nearest neighbor algorithm. Using these rankedlists, a user may advantageously be presented with an ordered list ofnearest, known safe threat samples and nearest, known unsafe samples. Ak-nearest neighbor algorithm is a non-parametric method that assigns anew item to a particular class based on a closest neighbor within a(usually multi-dimensional) features space for training data.

While this approach provides a computationally efficient technique forevaluating similarity for certain data types, it will be understood thatother computational measures of similarity are known in the art, and mayusefully be employed to evaluate similarity of a new threat sample toknown safe an unsafe threat samples as contemplated herein. For example,a nearest centroid classifier or nearest prototype classifier uses aclassification model that assigns a classification based on a closestcentroid that may be used to assess similarity as contemplated herein.As another example, an n-gram analysis supports efficient approximatematching and may be used to perform fast, large scale similarityanalysis for a given file path over a large database of known maliciousand known benign file paths and URLs.

While certain portions of this description emphasize the analysis ofexecutables for detection of suspiciousness or the identification ofintermediate threats, it should be understood that the term “threatsample” is not so limited. Other threat samples based on, e.g., files,caches, or other data sources may be used. Events, e.g., in a filteredevent stream may also or instead be used, and the techniques describedherein for use with code samples are also generally applicable to otherthreat samples instead of explicit computer code such as networkactivity, content, event streams that identify activities or behaviors,and so forth. Thus for example, activities such as visiting a particularURL, opening an attachment, sending an electronic mail, or other eventsmay also or instead be analyzed as threat samples by an integrativemodel or other threat detection tools to identify potential malwarethreats on an endpoint or group of endpoints.

As shown in step 810, the method 800 may include displaying theintermediate threat(s) and supplemental information in a user interfacefor user disposition, or otherwise augmenting a description of the newthreat sample in a user interface with the supplemental information.This may, for example, include presenting a description of the newthreat sample, the one or more relevant features, and the similar threatsamples in a user interface. In one aspect, the method may includedisplaying a list of the similar threat samples ranked according tosimilarity to the new threat sample using, e.g., a k-nearest neighboralgorithm or any other suitable technique for measuring similarity. Thismay, for example, include similarity of executable code, similarity ofbehaviors, similarity of filenames, similarity of URL's called, orsimilarity of any other objective feature or combination of featuresthat can be correlated to risk (or lack of risk). In one aspect, anumber of the most similar safe samples and a number of the most similarunsafe samples may be presented together, and ranked, e.g., based onrelative threat or based on similarity. The threat samples may bedisplayed along with descriptive information, attributes, behavioralcharacteristics, metadata and so forth, as well as any other informationthat might help a human user assess relative similarity when disposingof the current, new threat sample.

More generally, any supplemental information that might be helpful to auser in assessing a new threat sample may usefully be gathered anddisplayed to the user. For example, this may include augmenting thedescription of the new threat sample with a reputation of the new threatsample, e.g., based on reputation information available from a threatmanagement facility. This may also or instead include augmenting thedescription of the new threat sample with a suspiciousness score basedon a genetic analysis of features of the new threat sample. In anotheraspect, this may include augmenting the description of the new threatsample with contextual information such as users, related processes,associated data sources or files used by the threat sample, signatureanalysis, behavioral analysis, software update history or status for theendpoint, and so forth.

As shown in step 812, the method 800 may include disposing of theintermediate threat(s), such as by receiving user input through the userinterface categorizing the new threat sample as safe, unsafe, orundetermined. Thus in one aspect, the user interface may be configuredto receive a user input categorizing the new threat sample as safe,unsafe or undetermined. Where a disposition as unsafe does notautomatically initiate a remedial action, the user interface may also beconfigured to receive an express instruction for a remedial action suchas any of the remedial actions described herein, or any other actionssuitable for disposing of or otherwise managing a new threat. In anotheraspect, the user interface may be configured to receive user input toadjust filtering of an event stream from an endpoint that provided thenew threat sample, which may permit an increase or decrease in theamount of event reporting from the endpoint instead of, or in additionto, a specific characterization of the new threat sample.

In another aspect, a system as contemplated herein includes a memorystoring a first model for evaluating a likelihood that a threat sampleis at least one of safe or malicious, a second model characterizing amanner in which a number of human-interpretable features contribute toan evaluation of suspiciousness of a file, and a third model forevaluating similarity of threat samples. The system may include a threatmanagement facility including a processor configured to apply the firstmodel to identify a new threat sample as an intermediate threat when thenew threat sample is not within a predetermined likelihood of beingmalicious or safe according to the first model. The system may alsoinclude a web server configured to present a user interface including adescription of the intermediate threat, augmented by one or morefeatures of the intermediate threat identified with the second model andone or more similar threat samples identified with the third model, theweb server further configured to receive input from a user through theuser interface disposing of the intermediate threat. Disposing of theintermediate threat may include remediating the intermediate threat.Disposing of the intermediate threat may also or instead includecharacterizing the intermediate threat as safe, unsafe or undetermined.

FIG. 9A shows a user interface for managing intermediate threats in anenterprise network. The user interface 900 may be provided, e.g., as aweb page or other content presented from the threat management facilityfor display on a user device such as an end user endpoint. The userinterface 900 may show a feed 902 of suspicious events. The eventswithin this feed 902 may be sorted, e.g., into files, URL visits,executables, processes, downloads, and so forth, or any other usefulcategories for review, or the events may be combined into a single feed.As noted above, threat samples may include executable code, however, thetechniques contemplated herein may also or instead be applied to threatsamples such as files, network activity, or streams of event data.

A variety of tools 904 for explicit disposition of new threat samplesmay be provided. For example, the user interface 900 may include tools904 such as buttons or similar controls for a user to mark a particularevent as, e.g., safe, unsafe, low priority, unknown or the like. Theuser interface 900 may also provide controls for querying the enterprisenetwork for additional information, for adjusting filtering of eventstreams from endpoint data recorders, for initiating scans or otheranalysis, and so forth.

In one aspect, the user interface 900 may display a window 906 with moregranular information about features contributing to suspiciousness. Forexample, an analysis of a threat sample may return a 90% suspicion ofmalicious code, while a file path analysis may return a 57% suspicion,and a URL analysis may return a 77% suspicion. While an integrativemodel may combine these various features into a single estimate ofsuspiciousness or potential risk, the individual values may be useful toa user attempting to manually dispose of an intermediate threat.Furthermore, for any particular feature (e.g., the URL analysis in FIG.9), a number of most similar events or threat samples for that featuremay be displayed, with similarity evaluated using, e.g., a k-nearestneighbor algorithm or other algorithm for evaluating similarity within afeature space. These more granular estimates of suspiciousness may bepresented in separate sub-windows, which may usefully be arranged in anaccordion, a stacked group of drop-down lists, or any other suitablecontrol element or combination of control elements that permits eachtype of estimate to be expanded or collapsed under user control.

FIG. 9B shows a user interface for managing intermediate threats in anenterprise network. The user interface 950 may, for example, include anyof the user interfaces described herein.

In one aspect, the user interface 950 may show a window 952 listinghuman interpretable features contributing to an estimate ofsuspiciousness. For example, the user interface 950 may presentparticular features in the window 952 such as whether a threat sample issigned, whether the threat sample calls cryptographic libraries, andwhether the threat sample inspects other processes. For each suchfeature, the user interface 950 may further present the number of knowngood and known bad threat samples for that feature, with the featuresprogressively nested according to the hierarchy of a random

The features displayed in this list may be a subset of features in arandom forest over human-interpretable features that is selected basedon relevance, e.g., how strongly indicative those features are of safetyor suspiciousness. In one aspect, this may include features that aremost heavily weighted on a percentage basis toward safety orsuspiciousness. In another aspect, this may include features with thelargest number of relevant samples (e.g., higher up the decision tree).In another aspect, these and any other factors may be weighted orotherwise collectively evaluated to select a subset of features fordisplay to a user. This approach may usefully assist a human user whenevaluating an intermediate threat for manual disposition by providing adisplay of features that contribute more significantly or mostsignificantly to the potential risk associated with a threat sample.

In another aspect, the user interface may provide a display of therandom forest output (e.g., quantitative data about varioushuman-interpretable features), or a display of most similar safe andunsafe threat samples, or some combination of these. For example, theuser interface may provide one or more user controls for the user toselect among these different analyses, and/or other analyses, contextualinformation, or other supplemental information.

FIG. 10 shows a user interface for managing intermediate threats in anenterprise network. In general, the user interface 1000 may include amap 1002 of the genetic composition of an intermediate threat sample1004 and similar sets of safe threat samples 1006 and unsafe threatsamples 1008. The map 1002 may show, for each of the known threatsamples, the presence and absence of a number of genetic features thatare present in the unknown, intermediate threat sample 1004. The geneticfeatures may be any features useful for characterizing threat samplesincluding, without limitation, behaviors, associated events, fileactivities, network activity, signatures, certificates, sourceinformation, file content, source code, context, and so forth. As asignificant advantage, this permits a visual assessment of behavioralsimilarity (or other genetic information) by a human reviewerindependent of machine learning and other computerized analysis.

FIG. 11 shows a flow chart of a method for dynamic filtering of endpointevent streams. In general, activity on an endpoint is monitored in twostages with a local agent. In a first stage, particular computingobjects on the endpoint are selected for tracking. In a second stage,particular types of changes to those objects are selected. By selectingobjects and object changes in this manner, a compact data stream ofinformation highly relevant to threat detection can be provided from anendpoint to a central threat management facility. In order to supportdynamic threat response, the locus and level of detection applied by thelocal agent can be controlled by the threat management facility.

As shown in step 1102, the method 1100 may include instrumenting theendpoint, e.g. with a local agent, to detect a plurality of types ofchanges to a plurality of computing objects. In general, the changes maybe any of the events or other actions described herein, and thecomputing objects may be any of the computing objects described herein.For example, the computing objects may include a number of files, anumber of processes, and/or a number of executables. The computingobjects may also or instead include one or more of an electroniccommunication, a registry of system settings, a secure kernel cache, orany other data or data structure stored on an endpoint or communicatedto or from the endpoint. Similarly, the types of changes may be anytypes of changes that might usefully be monitored in a threat managementcontext as contemplated herein. For example, the endpoint may beinstrumented to detect file reads and writes, but not file opens orcloses. Or the endpoint may be instrumented to monitor inbound andoutbound electronic mail, but not outbound electronic mail to otherusers within the enterprise. As another example, the endpoint may beinstrumented to monitor changes to operating system registry entries bynon-system processes, or to monitor read/write activity thatsubstantially increases file entropy. More generally, any types ofchanges that might contribute to a determination of suspiciousness orsafety can usefully be monitored, with instrumentation of suitable,corresponding computing objects, all as contemplated herein.

As shown in step 1104, the method 1100 may include creating an eventstream from the local agent including each type of change to each of thecomputing objects detected on the endpoint.

As shown in step 1106, the method 1100 may include storing the eventstream in a data recorder on the endpoint. This may generally be anunfiltered event stream containing additional event data not includingin a filtered event stream that is sent to a threat management facility,and may include some or all of the event data that the endpoint isinstrumented to detect. For example, the unfiltered event stream mayinclude additional ones of the plurality of types of changes to theplurality of computing objects in a filtered event stream, or changes toadditional ones of the plurality of computing objects not included inthe filtered event stream.

As shown in step 1108, the method 1100 may include processing the eventstream with a filter at the endpoint to provide a filtered event streamincluding a subset of the types of changes to a subset of the computingobjects. In one aspect, the subset of computing objects includes one ormore of a file, an executable, a process, a database, and a message. Inanother aspect, the types of changes include at least one of a fileread, a file write, a file copy, a file encrypt, a file decrypt, anetwork communication, a registry update, a software installation, achange in permissions, and a query to a remote resource. It will beunderstood that, while the filtered event stream is illustrated asflowing from the event stream stored by the data recorder, the filteredevent stream may also or instead be created directly by a security agentas the unfiltered event stream is captured and forwarded to the datarecorder for storage.

Processing the event stream with the filter may also include locallyadjusting the filter at the endpoint, e.g., in response to local changesdetected on or by the endpoint. For example, the level of filtering maybe locally adjusted by the endpoint based on a reputation score for oneor more processes, files or the like on the endpoint. This filtering maybe done for all detectable events on the endpoint, or for specificprocesses. Thus, for example, when a reputation for a new process orother computing object is unknown, the endpoint may decrease filteringto provide greater data reporting to the threat management facility forthat particular process. Thus, while step 1116 below contemplatescontrolling the filter from a central threat management facility or thelike, the filter may also or instead be controlled locally on anendpoint in response to changes in security posture, policy complianceposture, or any other events, context, malware detections, and so forth.

In one aspect, the filtered event stream may be arranged around anchorpoints such as a file, a domain name, or any other useful piece of dataor metadata for which the presence can be monitored on an endpoint. Forexample, a file hash may be created for a file and used to test for thepresence of that file on endpoints throughout an enterprise. Wheneverthis anchor point, e.g., the corresponding file hash, is detected on anendpoint, a collection of related events, metadata, context and so forthmay be added to the filtered event stream for reporting to a centralthreat management facility.

In another aspect, the level of filtering may be locally controlledbased on factors or requirements other than threat detection. Forexample, an event stream may be filtered to remove personal identifyinginformation, e.g., for compliance with data privacy regulations. Asanother example, filtering may be controlled based on network usagerestrictions, e.g., so that a particular endpoint does not exceed apredetermined hourly, daily, or weekly quota of bandwidth for eventreporting.

Further, it will be understood that the filtered event stream mayinclude synthetic events that characterize other collections of eventsin a single event or condensed group of events. This approachadvantageously permits more compact communication of relevantinformation to a threat management facility, as well as more compactstorage of information on the endpoint. In one aspect, the syntheticevents may be stored by the data recorder in place of (e.g., to reducememory requirements) or in addition to (e.g., to reduce communicationsrequirements while preserving a more complete log or related activity)more detailed logging of granular events on the endpoint. In anotheraspect, the data recorder may store complete event details, and theendpoint may (e.g., with the security agent) create synthetic eventsdynamically to facilitate more compact communication to the threatmanagement facility.

As shown in step 1110, the method 1100 may include transmitting thefiltered event stream to a threat management facility. The filteredevent stream may be transmitted at any suitable frequency includingperiodic, aperiodic or other scheduled transmittal, as well as pushedtransmittal (e.g., at intervals determined by the endpoint) or pulledtransmittal (e.g., at intervals determined by the threat managementfacility, or any combination of these. Thus, for example, the endpoint(or security agent on the endpoint) may periodically report the filteredevent stream on a predetermined schedule, with supplemental transmittalsprovided when the security agent detects a potential threat, orrequested when the threat management facility detects a potentialthreat.

As shown in step 1112, the method 1100 may include receiving thefiltered event stream at the threat management facility.

As shown in step 1114, the method 1100 may include processing thefiltered event stream at the threat management facility to evaluate asecurity state of the endpoint. This may include any processing suitablefor analyzing the events within the filtered event stream. For example,processing the filtered event stream may include searching for potentialmalicious activity on the endpoint, e.g., based on a pattern ofactivities within the filtered event stream, or based on a specificactivity such as an unauthorized change to a registry entry. Processingthe filtered event stream may also or instead include searching for asecurity exposure on the endpoint such as a missing security patch, achange in a firewall configuration, a de-installation of a malwarescanner, and so forth. In another aspect, processing the filtered eventstream may include securely verifying a status of the endpoint, e.g.,with a secure heartbeat or the like from the endpoint, in order toensure that the endpoint has not been otherwise compromised. In anotheraspect, processing the filtered event stream may include monitoring forchanges that bring the endpoint out of compliance with a security policyfor an enterprise, or otherwise present an actual or potential risk tonetwork security for the enterprise.

As shown in step 1116, the method 1100 may include conditionallytransmitting adjustments to filtering by the endpoint. For example, themethod 1100 may include, in response to a predetermined security statedetected by the threat management facility, transmitting an adjustmentto the endpoint for at least one of the types of changes or thecomputing objects used by the filter to process the event stream. Thismay include transmitting an adjustment to a filter used by the endpointto select which of the plurality of types of changes to the plurality ofcomputing objects the data recorder reports in the filtered eventstream. Thus, for example, when the security state indicated by thefiltered event stream is a potentially compromised state of a file,process or the like, the threat management facility may decreasefiltering in order to receive more data about various changes to or bycomputing objects on the endpoint. This may include general changes tothe level of filtering, or targeted changes that focus on specificcomputing objects or types of changes that might be related to apotential compromise. In one aspect, the adjustment to endpointfiltering may include a change to the subset of types of changesincluded in the filtered event stream, such as by increasing the typesof changes included in the filtered event stream when the endpoint ispotentially compromised, or decreasing the types of changes included inthe filtered event stream when a potential compromise has beenremediated. The adjustment may also or instead include a change to thesubset of computing objects included in the event stream, such as bymonitoring additional processes, directories or the like when apotential compromise is detected.

Adjustments may also be made to filtering by other endpoints within anenterprise network. For example, where a compromise is detected on oneendpoint, behaviors or other patterns detected in the (filtered) eventstream for that endpoint may be used to adjust the filtering on otherendpoints to facilitate the detection of similar or related patternselsewhere within the enterprise network. Similarly, endpoints or dataresources known to contain high business value assets may have filteringadjusted to facilitate more detailed and frequent monitoring of relatedassets.

In another aspect, filtering may be adjusted independently of thecurrent filtered event stream, e.g., based on other context. Forexample, when an employee is about to leave a company, filtering may bereduced on or removed from any associated compute instances so thatcomputing or network activity can be more closely monitored untildeparture.

As shown in step 1118, the method 1100 may include other processingbased on the filtered event stream. For example, the method 1100 mayinclude correlating the filtered event stream to a malware event on theendpoint and searching for the malware event on one or more otherendpoints coupled to the enterprise network based on a pattern of eventsin the filtered event stream. In another aspect, the method 1100 mayinclude storing the filtered event stream at the threat managementfacility. In another aspect, the method 1100 may include, when thefiltered event stream shows that the security state of the endpoint iscompromised, initiating a remedial action, e.g., using any of theremediation tools available to the threat management facility.

According to the foregoing, there is also disclosed herein a systemincluding an endpoint and a threat management facility. The endpoint mayexecute a data recorder to store an event stream including a pluralityof types of changes to a plurality of computing objects detected on theendpoint, and the endpoint may execute a local agent to process theevent stream with a filter into a filtered event stream including asubset of the plurality of types of changes to a subset of the pluralityof computing objects. The local agent may be further configured tocommunicate the filtered event stream to a remote resource over a datanetwork. The threat management facility may be configured to receive thefiltered event stream from the endpoint and to process the filteredevent stream to evaluate a security state of the endpoint. The threatmanagement facility may be further configured to respond to apredetermined change in the security state by transmitting an adjustmentto the endpoint for at least one of the types of changes or thecomputing objects used by the filter to process the event stream. In oneaspect, the threat management facility may be configured to initiate aremediation of the endpoint when the security state of the endpoint iscompromised.

FIG. 12 shows a flow chart of a method for forensic query of local eventstreams in an enterprise network. In general, activity on an endpoint ismonitored in two stages with a local agent. In a first stage, particularcomputing objects on the endpoint are selected for tracking. In a secondstage, particular types of changes to those objects are selected. Byselecting objects and object changes in this manner, a compact datastream of information highly relevant to threat detection can beprovided from an endpoint to a central threat management facility. Atthe same time, a local data recorder creates a local record of a widerrange of objects and changes. The system may support forensic activityby facilitating queries to the local data recorder on the endpoint toretrieve more complete records of local activity when the compact datastream does not adequately characterize a particular context.

As shown in step 1202, the method 1200 may include instrumenting theendpoint as described herein, e.g. with a local agent, to detect aplurality of types of changes to a plurality of computing objects. Ingeneral, the changes may be any of the events or other actions describedherein, and the computing objects may be any of the computing objectsdescribed herein. For example, the computing objects may include anumber of files, a number of processes, and/or a number of executables.The computing objects may also or instead include one or more of anelectronic communication, a registry of system settings, and a securekernel cache.

As shown in step 1204, the method 1200 may include creating an eventstream from the local agent including, for example, each type of changeto each of the computing objects detected on the endpoint.

As shown in step 1206, the method 1200 may include storing the eventstream in a data recorder on the endpoint. As described above, this maygenerally be an unfiltered event stream containing additional event datanot including in a filtered event stream that is sent to a threatmanagement facility, such as some or all of the event data that theendpoint is instrumented to detect. For example, the unfiltered eventstream may include additional ones of the plurality of types of changesto the plurality of computing objects in a filtered event stream, or oneor more of the plurality of types of changes to additional ones of theplurality of computing objects.

As shown in step 1208, the method 1200 may include processing the eventstream with a filter at the endpoint to provide a filtered event streamincluding a subset of the types of changes to a subset of the computingobjects. In one aspect, the subset of computing objects includes one ormore of a file, an executable, a process, a database, and a message. Inanother aspect, the types of changes include at least one of a fileread, a file write, a file copy, a file encrypt, a file decrypt, anetwork communication, a registry update, a software installation, achange in permissions, and a query to a remote resource.

As shown in step 1210, the method 1200 may include transmitting thefiltered event stream to a threat management facility, e.g., asdescribed above.

As shown in step 1212, the method 1200 may include receiving thefiltered event stream at the threat management facility.

As shown in step 1214, the method 1200 may include processing thefiltered event stream at the threat management facility to evaluate asecurity state of the endpoint. This may include any processing suitablefor the events within the filtered event stream. For example, processingthe filtered event stream may include searching for potential maliciousactivity on the endpoint, e.g., based on a pattern of activities withinthe filtered event stream, or based on a specific activity such as anunauthorized change to a registry entry. Processing the filtered eventstream may also or instead include searching for a security exposure onthe endpoint such as a missing security patch, a change in a firewallconfiguration, a de-installation of a malware scanner, and so forth. Inanother aspect, processing the filtered event stream may includesecurely verifying a status of the endpoint, e.g., with a secureheartbeat or the like from the endpoint, in order to ensure that theendpoint has not been otherwise compromised. More generally, this mayinclude any of the processing described herein that might usefully beperformed by a threat management facility based on an event stream fromone or more endpoints associated with an enterprise network.

As shown in step 1216, the method 1200 may include conditionallytransmitting a request to the endpoint, or more specifically, the datarecorder on the endpoint, for additional event data in the unfilteredevent stream. For example, this may include, in response to apredetermined security state detected by the threat management facility,requesting additional event data from the data recorder for at least oneof other ones of the types of changes than the subset of the types ofchanges or other ones of the plurality of computing objects than thesubset of the computing objects. The request may include a request forall event data in an unfiltered event stream stored by the data recorderover a predetermined time window. The request may also or insteadinclude a request for a larger group of types of changes or events fromadditional computing objects. The predetermined change in the securitystate may be any change raising suspicion or otherwise indicating thatadditional information may be useful for manual review, automatedreview, forensic documentation, or some combination of these. Forexample, the predetermined change in the security state of the endpointmay include an increased likelihood of malicious activity associatedwith the endpoint. The change may also or instead include a change inpolicy compliance, detection of known malware, suspicious networkcommunications, access to highly valuable business assets, and so forth.

As shown in step 1218, the method 1200 may include other processingbased on the filtered event stream. For example, the method 1200 mayinclude correlating the filtered event stream to a malware event on theendpoint and searching for the malware event on one or more otherendpoints coupled to the enterprise network based on a pattern of eventsin the filtered event stream. In another aspect, the method 1200 mayinclude storing the filtered event stream at the threat managementfacility. In another aspect, the method 1200 may include, when thefiltered event stream shows that the security state of the endpoint iscompromised, initiating a remedial action, e.g., using any of theremediation tools available to the threat management facility. Moregenerally, any action necessary or helpful for detecting, investigating,disposing of, or otherwise managing threats based on the filtered eventstream may usefully be performed in this step.

According to the foregoing, in one aspect, there is disclosed herein asystem including an endpoint and a threat management facility. Theendpoint may execute a data recorder to store an event stream of eventdata including a plurality of types of changes to a plurality ofcomputing objects detected on the endpoint. The endpoint may alsoexecute a local agent configured to process the event stream with afilter into a filtered event stream including a subset of the pluralityof types of changes to a subset of the plurality of computing objects.The local agent may be further configured to communicate the filteredevent stream to a remote resource over a data network. The threatmanagement facility may be configured to receive the filtered eventstream from the endpoint and to process the filtered event stream toevaluate a security state of the endpoint, the threat managementfacility further configured to respond to a predetermined change in thesecurity state by transmitting a request to the endpoint for additionalevent data stored by the data recorder. In one aspect, the threatmanagement facility is further configured to initiate a remediation ofthe endpoint when the security state of the endpoint is compromised.

FIG. 13 shows a flow chart of a method for threat detection withbusiness impact scoring. In general, a computer model is created forautomatically evaluating the business value of computing objects such asfiles and databases on an endpoint. This can be used to assess thepotential business impact of a security compromise to an endpoint, or aprocess executing on an endpoint, in order to prioritize potentialthreats within an enterprise for human review and intervention.

As shown in step 1302, the method 1300 may include providing a valuationmodel for automatically estimating a business value of a file. Providingthe valuation model may, for example, include training a machinelearning algorithm to estimate the business value based on a trainingset of files each having a known business value. This may includetraining a machine learning model to recognize files with (known) highbusiness value based on, e.g., ownership, authorship, content, accesscontrols, and so forth. For example, the model may be trained torecognize credit card numbers, social security numbers, or othersensitive information including financial information, personalinformation, and other sensitive content within files indicative ofactual or potential business value. The model may also or instead betrained to recognize potentially sensitive documents based on documenttype. For example, the model may be trained to classify documents aspatent applications, resumes, financial statements, bank statements andso forth, with the corresponding classification used to assign anestimated value as appropriate.

This may also or instead include providing rules, regression models,heuristics, and so forth for identifying high business value files orotherwise estimating the value of files, data, content and the like. Thevaluation model may, for example, estimate value based on file location,based on an access control content, based on content, or based on anyother context, usage, feature or combination of the foregoing. Forexample, the valuation model may estimate value based on one or more ofencryption status, file type, file usage history, file creation date,file modification date, file content, and file author. More generally,this may include any human-interpretable features, or any other featuresuseful for estimating business value, human-interpretable or otherwise,such as features independently identified by a clustering algorithm orother unsupervised machine learning technique.

These techniques may also or instead be used to estimate the businessvalue of a machine or other domain based on the aggregated businessvalue of files and the like within that estate. Thus while thedescription herein focuses on business value on a file-by-file basis,the method 1300 contemplated herein may also or instead by used on amachine-by-machine basis or any other basis to estimate the businessimpact of potent threats.

As shown in step 1304, the method 1300 may include providing anintegrative model, such as any of the integrative models describedherein. For example, this may include creating an integrative model thatevaluates a potential threat by a threat sample based on a combinationof a first model configured to identify malicious code based onbehavioral tags, a second model configured to identify malicious codebased on an executable file path, and a third model configured toidentify malicious code based on a Uniform Resource Locator within thethreat sample. More generally, the integrative model may evaluatepotential threats by computer objects based on one or more of filebehavior, file signature, file path, Uniform Resource Locators accessed,or any other feature or combination of features suitable for assessingsuspiciousness. The integrative model may also or instead include one ormore machine learning models trained to recognize potentially maliciouscode based on a training set of known safe and known unsafe threatsamples.

As shown in step 1306, the method 1300 may include identifyingintermediate threats, e.g., with the integrative model. The one or moreintermediate threats may include one or more computing objects with anobjective score from the integrative model that are not within apredetermined confidence level of a safe score or a malicious score. Theone or more computing objects may, for example, include a process, anexecutable, a file, and so forth. The one or more computing objects mayalso or instead include a registry of system settings, a secure kernelcache of process information, or any other data source, cache, resourceor the like that might be usefully monitored for threat detection ascontemplated herein.

Identifying intermediate threats may, for example, include configuring athreat management facility to evaluate new threat samples on endpointswithin an enterprise network according to the integrative model basedon, e.g., a filtered event stream as described herein, or any othertechnique or combination of techniques suitable for identifying code orother threat samples that cannot confidently be classified as safe orunsafe. Identifying intermediate threats may include evaluating newthreat samples, such as by identifying one or more intermediate threatsby any of the new threat samples that are not within a predeterminedconfidence level of safe code or malicious code according to theintegrative model. It will be appreciated that, while an integrativemodel as contemplated herein is one useful model for identifying codethat is not clearly safe or unsafe, other techniques for identifyingintermediate threats may also or instead be used.

As shown in step 1308, the method 1300 may include estimating a businessvalue of one or more intermediate threats with the valuation modeldescribed above. This may include generating an estimated dollar valueof the contents of files accessed by a process or other computingobject, or an estimated business impact of the public dissemination ofinformation contained in such files. This may also or instead includegenerating a score otherwise indicative of business value based on anyof the factors or features described herein.

As shown in step 1310, the method 1300 may include providing a userinterface for presenting the one or more intermediate threats to a userfor human evaluation. This may, for example, include any of the userinterfaces described herein.

As shown in step 1312, the method 1300 may include ranking the one ormore intermediate threats for presentation within the user interface,e.g., by ranking the intermediate threats with the valuation modeldescribed above. More generally, the intermediate threats may be rankedusing any technique that reflects actual or potential business impact ofthe threat based on business value of affected data or computeinstances, the likelihood or severity of the potential risk, or somecombination of these. Thus in one aspect, ranking the intermediatethreats may include ranking the intermediate threats based on acombination of a likelihood of maliciousness determined according to theintegrative model and an estimated business value of associated filesdetermined according to the valuation model.

As shown in step 1314, the method 1300 may include presenting a list ofthe one or more intermediate threats in the user interface. As discussedabove, the list may be ranked according to a combination of an objectivescore of riskiness or suspiciousness (e.g., from the integrative model)and an objective score for the business value (e.g., from the valuationmodel).

As shown in step 1316, the method 1300 may include receiving a userdisposition of an intermediate threat, for example using any of thetechniques described herein. For example, this may include receiving auser-initiated remedial action for one of the intermediate threats inthe user interface. This may also or instead include receiving a userrisk assessment for one of the intermediate threats in the userinterface, such as by explicitly categorizing the intermediate threat assafe, unsafe, unknown, or appropriate for increased monitoring. Inanother aspect, the method 1300 may include remediating a risk to a highbusiness value computing object in response to a user input in the userinterface.

According to the foregoing, there is disclosed herein a system includinga memory storing an integrative model and a valuation model, a threatmanagement facility, and a web server. The integrative model may beconfigured to evaluate a potential threat by a threat sample based on acombination of a first model configured to identify malicious code basedon behavioral tags, a second model configured to identify malicious codebased on an executable file path, and a third model configured toidentify malicious code based on a Uniform Resource Locator within thethreat sample, and the valuation model configured to estimate a businessimpact of the potential threat based on an estimated business value ofone or more files associated with the threat sample. The threatmanagement facility may be configured to apply the integrative model tonew threat samples and to identify intermediate threats that are notwithin a predetermined likelihood of being safe or unsafe. The webserver may be configured to display a list of intermediate threats in auser interface, wherein the list of intermediate threats is rankedaccording to a combination of a first score from the integrative modeland a second score from the valuation model. In one aspect, the threatmanagement facility may be configured to remediate a risk to an endpointin response to a user input received through the user interface.

The above systems, devices, methods, processes, and the like may berealized in hardware, software, or any combination of these suitable fora particular application. The hardware may include a general-purposecomputer and/or dedicated computing device. This includes realization inone or more microprocessors, microcontrollers, embeddedmicrocontrollers, programmable digital signal processors or otherprogrammable devices or processing circuitry, along with internal and/orexternal memory. This may also, or instead, include one or moreapplication specific integrated circuits, programmable gate arrays,programmable array logic components, or any other device or devices thatmay be configured to process electronic signals. It will further beappreciated that a realization of the processes or devices describedabove may include computer-executable code created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software. In another aspect, themethods may be embodied in systems that perform the steps thereof, andmay be distributed across devices in a number of ways. At the same time,processing may be distributed across devices such as the various systemsdescribed above, or all of the functionality may be integrated into adedicated, standalone device or other hardware. In another aspect, meansfor performing the steps associated with the processes described abovemay include any of the hardware and/or software described above. Allsuch permutations and combinations are intended to fall within the scopeof the present disclosure.

Embodiments disclosed herein may include computer program productscomprising computer-executable code or computer-usable code that, whenexecuting on one or more computing devices, performs any and/or all ofthe steps thereof. The code may be stored in a non-transitory fashion ina computer memory, which may be a memory from which the program executes(such as random-access memory associated with a processor), or a storagedevice such as a disk drive, flash memory or any other optical,electromagnetic, magnetic, infrared or other device or combination ofdevices. In another aspect, any of the systems and methods describedabove may be embodied in any suitable transmission or propagation mediumcarrying computer-executable code and/or any inputs or outputs fromsame.

It will be appreciated that the devices, systems, and methods describedabove are set forth by way of example and not of limitation. Absent anexplicit indication to the contrary, the disclosed steps may bemodified, supplemented, omitted, and/or re-ordered without departingfrom the scope of this disclosure. Numerous variations, additions,omissions, and other modifications will be apparent to one of ordinaryskill in the art. In addition, the order or presentation of method stepsin the description and drawings above is not intended to require thisorder of performing the recited steps unless a particular order isexpressly required or otherwise clear from the context.

The method steps of the implementations described herein are intended toinclude any suitable method of causing such method steps to beperformed, consistent with the patentability of the following claims,unless a different meaning is expressly provided or otherwise clear fromthe context. So, for example, performing the step of X includes anysuitable method for causing another party such as a remote user, aremote processing resource (e.g., a server or cloud computer) or amachine to perform the step of X. Similarly, performing steps X, Y and Zmay include any method of directing or controlling any combination ofsuch other individuals or resources to perform steps X, Y and Z toobtain the benefit of such steps. Thus, method steps of theimplementations described herein are intended to include any suitablemethod of causing one or more other parties or entities to perform thesteps, consistent with the patentability of the following claims, unlessa different meaning is expressly provided or otherwise clear from thecontext. Such parties or entities need not be under the direction orcontrol of any other party or entity, and need not be located within aparticular jurisdiction.

It should further be appreciated that the methods above are provided byway of example. Absent an explicit indication to the contrary, thedisclosed steps may be modified, supplemented, omitted, and/orre-ordered without departing from the scope of this disclosure.

It will be appreciated that the methods and systems described above areset forth by way of example and not of limitation. Numerous variations,additions, omissions, and other modifications will be apparent to one ofordinary skill in the art. In addition, the order or presentation ofmethod steps in the description and drawings above is not intended torequire this order of performing the recited steps unless a particularorder is expressly required or otherwise clear from the context. Thus,while particular embodiments have been shown and described, it will beapparent to those skilled in the art that various changes andmodifications in form and details may be made therein without departingfrom the spirit and scope of this disclosure and are intended to form apart of the invention as defined by the following claims, which are tobe interpreted in the broadest sense allowable by law.

What is claimed is:
 1. A computer program product comprising computerexecutable code embodied in a non-transitory computer readable mediumthat, when executing on one or more computing devices, performs thesteps of: providing a model for evaluating a likelihood that a threatsample is at least one of safe or malicious based on a training set ofknown threat samples; identifying a new threat sample as an intermediatethreat that is not within a predetermined likelihood of being maliciousor safe according to the model; identifying one or more relevantfeatures of the new threat sample associated with an inference ofmalicious code using a random forest over human-interpretable featuresof the training set of known threat samples; identifying similar threatsamples including one or more safe threat samples similar to the newthreat sample and one or more malicious threat samples similar to thenew threat sample based on a k-nearest neighbor algorithm; presenting adescription of the new threat sample, the one or more relevant features,and the similar threat samples in a user interface; and receiving userinput through the user interface categorizing the new threat sample assafe, unsafe, or undetermined.
 2. The computer program product of claim1 wherein the similar threat samples include a list of safe threatsamples ranked based on similarity to the new threat sample according tothe k-nearest neighbor algorithm.
 3. The computer program product ofclaim 1 wherein the similar threat samples include a list of maliciousthreat samples ranked based on similarity to the new threat sampleaccording to the k-nearest neighbor algorithm.
 4. The computer programproduct of claim 1 wherein the model includes an integrative model thatevaluates a potential threat by a threat sample based on a combinationof a first model configured to identify malicious code based onbehavioral tags, a second model configured to identify malicious codebased on an executable file path, and a third model configured toidentify malicious code based on a Uniform Resource Locator within thethreat sample.
 5. A method comprising: providing a model for evaluatinga likelihood that a threat sample is at least one of safe or maliciousbased on a training set of known threat samples; identifying a newthreat sample as an intermediate threat that is not within apredetermined likelihood of being malicious or safe according to themodel; identifying supplemental information relevant to evaluation ofthe new threat sample, the supplemental information including relevantfeatures of the new threat sample contributing to an inference ofmalicious code; and augmenting a description of the new threat sample ina user interface with the supplemental information, the user interfaceconfigured to receive a user input categorizing the new threat sample assafe, unsafe or undetermined.
 6. The method of claim 5 wherein the modelincludes an integrative model that evaluates a potential threat by athreat sample based on a combination of a first model configured toidentify malicious code based on behavioral tags, a second modelconfigured to identify malicious code based on an executable file path,and a third model configured to identify malicious code based on aUniform Resource Locator within the threat sample.
 7. The method ofclaim 6 wherein providing the model includes training a machine learningmodel to identify malicious code in a training set including threatsamples that are known to be safe and known to be malicious.
 8. Themethod of claim 5 wherein identifying supplemental information includesidentifying one or more features using a random forest overhuman-interpretable features associated with an inference of maliciouscode.
 9. The method of claim 5 wherein identifying supplementalinformation includes identifying similar threat samples known to be safeor malicious.
 10. The method of claim 9 wherein identifying similarthreat samples includes identifying one or more safe threat samples mostsimilar to the new threat sample based on a k-nearest neighboralgorithm.
 11. The method of claim 9 wherein identifying similar threatsample includes identifying one or more malicious threat samples mostsimilar to the new threat sample based on a k-nearest neighboralgorithm.
 12. The method of claim 9 further comprising displaying alist of the similar threat samples ranked according to similarity to thenew threat sample.
 13. The method of claim 9 further comprisingaugmenting the description of the new threat sample with a reputation ofthe new threat sample.
 14. The method of claim 9 further comprisingaugmenting the description of the new threat sample with asuspiciousness score based on a genetic analysis of features of the newthreat sample.
 15. The method of claim 9 further comprising augmentingthe description of the new threat sample with contextual information.16. The method of claim 5 further comprising receiving a user inputthrough the user interface characterizing the new threat sample as safe,unsafe or undetermined.
 17. The method of claim 5 wherein the userinterface is further configured to receive user input to adjustfiltering of an event stream from an endpoint that provided the newthreat sample.
 18. A system comprising: a memory storing a first modelfor evaluating a likelihood that a threat sample is at least one of safeor malicious, a second model characterizing a manner in which a numberof human-interpretable features contribute to an evaluation ofsuspiciousness of a file, and a third model for evaluating similarity ofthreat samples; and a threat management facility including a processorconfigured to apply the first model to identify a new threat sample asan intermediate threat when the new threat sample is not within apredetermined likelihood of being malicious or safe according to thefirst model; and a web server configured to present a user interfaceincluding a description of the intermediate threat, augmented by one ormore features of the intermediate threat identified with the secondmodel and one or more similar threat samples identified with the thirdmodel, the web server further configured to receive input from a userthrough the user interface disposing of the intermediate threat.
 19. Thesystem of claim 18 wherein disposing of the intermediate threat includesremediating the intermediate threat.
 20. The system of claim 18 whereindisposing of the intermediate threat includes characterizing theintermediate threat as safe, unsafe or undetermined.